Milford is my go-to recommendation for young families. I lived in the Devon section for a while but moved to a bigger house in Orange, which is also a great town.

Milford has lots of family activities and a great downtown. Unfortunately both Milford and Orange do not have much housing inventory. Good luck!

Orange and Milford probably fit best but inventory, as you noted, is low. North Haven would be my second choice.

Orange shares a school district with Woodbridge and Bethany. Also great towns. A bit less convenient to the shops and restaurants you listed, but they are quieter.