I've only read the book once so my thoughts on this may be more from the movie version, but I got the feeling that no one else thought he was hideous except him. Like to everyone else he was a normal dude with a little bit of a lisp and it was his own trauma that made him zero in on whatever "flaws" he had as a way to make himself feel like a piece of shit.