I have the Fiio FW5 with physical buttons and it's substantially more annoying than the Samsung or Sony buds I have with touch controls, they are simultaneously harder to deliberately press and easier to accidently press while pushing in. Physical buttons require more force, too, which tends to dislodge the buds. It's bad enough I don't use them on the Fiio while I use them on the Samsung and Sony all the time, and they pretty much "just work".

One thing I have found is the big brands (Samsung, Sony) do seem much better with the touch controls than Chi-Fi earbud makers. Once you know where the touch surface is and how to hold them (usually by the sides), I literally never trigger my Samsung or Sony buds putting them in or adjusting them.