We have a smaller mortgage by hundreds of dollars a month, and larger income of over 10k/mo total- And it’s STILL way too tight and was probably a bad idea.

How many years are left on it? If that’s a relatively new 30 year, yes, people are right and you need to downsize.