Your dataset is rather small and it seems you are not really doing Augmentations? I would try different Augmentations, that should improve your results regardless of the used model. Have you looked at some frameworks for medical image segmentation? nnUNet comes to mind which would give you a solid baseline. How good are your results currently?

Is there a particular reason why you chose a patch size of 32x32? You have a rather shallow U-Net with this small patch size and not that much information in your patches but look rather uniform. I would try to go for maybe 128x128 or even 256x256. For 2d segmentation that should still not be too memory intensive. What's the batch size you are using? If you use for example a batch size of 32 you could force in the data loader that at least 8 of the scans should have some mass in them. Just play around a bit with this number to see how it works. Keep an eye on recall and precision in addition to dice to see if your false positives will rise.

Could you give a few more details on what kind of images you have, what you are trying to segment, your model...? Are you calculating you dice score and dice loss on foreground and background? Its usually a good idea to calculate it on the foreground and if you have more than one foreground class take the mean. That should already help a lot with class imbalance. Also I would add cross entropy or focal loss in addition to dice loss, that's something I have found to work well in general. You can also modify your data loader such that it will oversample foreground during training (say you have a batch size of 2 and force that at least one image has foreground). It's probably also a good idea to find a good baseline to compare against so you get a better idea how your performance is.

I feel like I've read a paper where they do something similar to this but I'm not completely sure. I'll try finding it.

Edit: maybe remind me of that in case I forget about it