My suggestion is using 8 bit or 4 bit quantization. Also you can using automatic device mapping on Transformers that can offload partially to your CPU (warning : It use lots of System Memory [RAM]).

There is some tricks that could increase your accuracy:

1. Adjust the parameter tuning more. Sometimes increasing or decreasing the Augmentation Value (Rotation, zoom , shear,etc) could improve accuracy
2. Unfreeze Batch Normalization Layers could help
3. Check for each class of the dataset if it's balanced or not. If the datasets is inbalanced, try to use ViT or SMOTE algorithm.
4. Increasing Dropout value , sometimes could help
5. After doing transfer learning, add pooling layers , either MaxPool or Average Pool.

Also I don't think you should dropout some networks before the Dense. Because Flatten is just making it the matrices into one dimensional (Flatten out).