Politely tell them that there are times that talking isn't the best option and avoiding small talk is uncomfortable for you. Then ask how they'd like you to inform them indirectly that you want to end the conversation.

Something confusing like that should do the trick.