So in headphones, yes frequency response at your eardrum (not what is seen on a FR graph), is in theory everything that matters. Given that distortion is below the audible threshold, and the headphones are operating in minimum phase.

The frequency response of a headphone affects the sound waves that are generated, and how they are shaped when combined. When it comes to "Technicalities", yes it can be linked back to FR, but not simply as something that you can point to one area and pick out. More how different frequencies are emphasized and what is affected by auditory masking.

That aside, there are a lot of outside factors that can affect what we hear or perceive, such as how the headphones feel, how spacious the cups are or how isolating they are.

Our senses are a mix of "simple" systems, that are filtered through our brain and can be changed by other senses, or interpretation