We also have to wait for the improvements by using fp8 kicks in. NVIDIA has recently published a paper demonstrating that it is feasible to train with fp8 and the new tensor cores are compaitble with that format. Just the software isn't there yet.