Looking up the origins of that term is a quick, slightly intriguing read. In past tense you can say somebody pissed-off (went away) and it have nothing to do with your own emotion.

But the act of telling someone to piss off, is indeed typically connected to being angry with said person. Oh, English.