Going to add something to this excellent answer:

If you can't specifically figure out why they're annoying so you can tell them a reason, the problem just might be you.

I've been annoyed at people because I had a bad night's sleep, or I heard a story four times by other people in the past day and they're telling it for the fifth time, or out of silly displaced anger, or for other reasons that were more to do with me than with them.

Before responding to their question, a good question to ask yourself is "are they annoying just to me?".

If you're not confident that the answer to that question is "no", odds are actually pretty good that you're the one that needs to work on the situation, not them.