It’s a question of amplitude. In theory, a single driver is optimal, since it’s a point source. Creating a driver capable of loudspeaker sound pressure levels, that can reproduce the entire human hearing frequency range with high resolution and low distorsion is tricky (almost impossible). Doing the same at headphone audio levels is much much more doable, probably even easier than overcoming the problems with multiple drivers and extremely short listening distances. With multiple drivers you also have crossovers, with their problems and added distorsions and phase issues.

Also, you have technologies as planar magnetics or electrostatics, that are very difficult to use well at loudspeaker volumes, but extremely viable at headphone levels. (Electrostatic designs are also an example of a seriously high-performing single driver loudspeaker, albeit one that requires BIG speakers if you want full range bass reproduction at somewhat respectable levels.

In the loudspeaker world, some companies are jumping through a substantial amount of hoops just to get speakers to behave like a single point source even though constrained by the need to use multiple drivers, like Genelec TheOnes (three-way coaxial point-source design with substantial audio and DSP trickery...).

Most (all?) high end headphones are single-driver designs. If there would be substantial advantages to instead using multiple drivers I'm sure many high-end headphone designs would, but they don't...