Have you tried using the segmentation models from the SMP package (Iakubovskii, P. (2019)? I built a segmentation model for dermoscopy images and pre-trained models consistently outperformed anything else, architecture didn't matter that much. I got best results with "U-Net with SegFormer pre-trained encoder".

It depends how much training data you have, but unless you have millions of samples, pre-training usually trumps architecture.