I believe 401k loans can be optimal when three conditions are met:

You’re not required to pay it back at once should you lose your job.

You still get your normal contributions and employer match.

You take it right before a market downturn.

I would add your monthly 401k loan repay amount to your monthly mortgage bill and compare that to your mortgage alone under PMI.

You obviously can’t time the market, but if a 401k loan will secure you a net monthly savings compared to a mortgage without a big down payment, it’s definitely an option. The maximum payback time on a 401k loan is five years, and that’s just not a significant dent in your retirement potential, and could even be a boon if the market timing happens to work out.

You are trading one investment for another, after all, stocks for equity. My house has done much better than my 401k over the last five years.