I think it's a bit more nuanced than that- for example, the studies showing benefits for kids who socially transition only, with no medication, etc. If it was purely a "physical" situation, that wouldn't make a difference.

Speaking for myself, my personal case is certainly more physical than mental- I generally couldn't give a toss what people label me as in public, and don't generally need or request much in the way of validation. However, I'm in the minority, for a lot of folks it makes a real positive impact. I have ASD, though, so this pattern runs throughout most of my life in general, making it hard to compare my experience to the hypothetical average. I'm very rarely subject to any emotional impact from anything people outside my close circles do or say- neurotypical people in general are the opposite, often quite exquisitely attuned to others.

If I had to speculate, it would be that it's both a physical issue (tied to certain physical signs of various hormone shifts in utero) as well as a complex of mental symptoms that depends on environment, family upbringing, and broader culture. I've met some people who are more like me- it's about the physical symptoms and access to medication makes the pain recede. But, there are plenty of people for whom social validation is clearly a major component. This may simply correlate with measures of self-security or other vagaries of personality.

Edit: there's also the somewhat interesting situations wherein men receiving certain treatments for their prostate cancer develop gender dysphoria even though they are being treated as men still. Between social forces and physical factors, it seems that the physical factors win out, but that doesn't make them unimportant, just secondary.