Have you tried making the model smaller by turning into 16 bit floats instead of 32 bit floats? If it’s already 16 bit you could try 8 bit ints and see if the performance drop is acceptable. I think tensorflow and torch both have these options available.

Less simple option is changing the architecture to make it even smaller, there’s a variety of methods. Before doing that I’d have a look around to see what sort of tricks everyone else with the same goals as you are using.