Sound like you have a dream job as far as the arrangements go, with a decent if not spectacular pay.

You can find that a new job gives you a 20% raise but the work-life balance sucks.

Since your hubby is making about $160k (judging by your post) I don't think it's worth it, personally.