>But it really does beg the question how do you decide what is optimal? Or are we all just planning for the worst/longest case scenario?

the best way to go about this is to reverse engineer your life.

figure out your life expectancy based on published science literature (i.e. your current health ,medical history, etc etc)

then estimate how much you need per year during retirement and work backwards to how much you have to contribute per year to get to that final future value +/- 3% or some other margin of error

its not the most accurate, but its the best way to estimate retirement needs.