I'm not sure that's really an option either with current rates. Our $510k mortgage has a payment of $2500 including tax/insurance.

Even if we downsized to a $300k house at current rates (7%ish), the loan calculator says our payment would be somewhere in the ballpark of $2300. A $200 difference.

Even if it were $500/mo difference, it still doesn't seem worth giving up our property (which we love) for $500/mo considering we'd be getting into something worth $300-350k and getting of something worth $625k.