Perhaps dig deeper on activation functions, optimization algorithm, or step sizes. Try some alternatives.
If your domain images (and things that differentiate between classes) are very different than those in the pretrained network maybe it doesn't have the features you need.