You are missing a test split. It's common to pick the best validation checkpoint but you still want a test split (that's completely unseen during training/model picking) to test your model.
You also need to be careful with the metrics you look at to test your model because your classes are very imbalanced.
I think with this few images I can't afford having a test set. Also, I thought that since I have approximately 50 million pixels to classify in the validation dataset, and given that computer vision practicioners often don't have a test split, I don't really need a test split. Now I'm not sure.
Do you suggest doing cross validation with the training stopping mechanism "train for precisely the same number of steps I did in this run" or with "train using checkpointing and choosing the best checkpoint as I did in this run"?
I would take the best checkpoints (aka when the validation loss starts diverging from the training loss). Not the same number of steps because it can happen that the networks don't converge to a minima at the same time, some may be stuck somewhere for longer.
Latter_Security9389 t1_izirxj3 wrote
You are missing a test split. It's common to pick the best validation checkpoint but you still want a test split (that's completely unseen during training/model picking) to test your model.
You also need to be careful with the metrics you look at to test your model because your classes are very imbalanced.