Honestly, I’m not a financial advisor, but I think you’re completely fine. I was in a similar situation a few years back and I haven’t had any issues. At a certain point, you do need to consider your happiness.