The impact of training time is not linear so neither are the benefits of speeding up. For example, going from 1hr to 5 minutes would be useful for experimentation/early development phases. However once I am training a model for Production then 12 hours overnight is fine. I have other things to do to fill the time. I think what is useful for faster training is to be able to see that the model is converging.

Inference time is important up to a point but performance engineering is about steady optimisation over the whole system. You can reach a floor on one part like inference and still have work in network or cpu-bound stages.