Just say, "I decline to answer". Don't be a pushover. If you stand your ground and be firm with it, you don't have to justify yourself. You lose respect when you have to justify everything. It gives the inquirer room to further question you. Make a decision to not answer and stick with it. If it's your boss or coworker prying into your private life, you definitely don't have to say anything. You can just change the topic or say, "my private life is none of your business" and hold your ground. If it's work related, they need to give you a legitimate justification for asking and they should articulate why they are asking. If it's not a solid reason, just say "I need a better reason".