He assumes because the dude was looking at him like he had a problem before he stepped foot in the place. I can't say for sure it's racism but unless this dude has had a bad experience in the past with OP then it's kind of easy to call that dude didn't want him in there from the get go just by looking at the cover of his book.