Yeah, not sure why that was the case for you. I regularly flip between high and low gain, when swapping between a wide range of headphones, and I have not noticed any noise or distortion - despite how it can come across in measurements. To me, whatever it takes to get it to the right volume is fine, so long as the signal remains perceptibly clean; and so far, I have not had any complaints.

No matter, though. So long as you found something you're happy with, and enjoy using, then it's all good. :)