Have you already tried different variants of GAN for more stable training?

You are missing a test split. It's common to pick the best validation checkpoint but you still want a test split (that's completely unseen during training/model picking) to test your model.

You also need to be careful with the metrics you look at to test your model because your classes are very imbalanced.