That's literally the opposite of what's true.

You want a high dampening factor to avoid a change to the sound. The dampening factor is load impedance divided by output impedance.

So, higher headphone impedance (which is the load impedance) leads to a higher dampening factor.

Or in other words, you can get away with higher output impedance on the amp if you have headphones with higher load impedance.