Is there a particular reason why you chose a patch size of 32x32? You have a rather shallow U-Net with this small patch size and not that much information in your patches but look rather uniform. I would try to go for maybe 128x128 or even 256x256. For 2d segmentation that should still not be too memory intensive. What's the batch size you are using? If you use for example a batch size of 32 you could force in the data loader that at least 8 of the scans should have some mass in them. Just play around a bit with this number to see how it works. Keep an eye on recall and precision in addition to dice to see if your false positives will rise.