Orange and Milford probably fit best but inventory, as you noted, is low. North Haven would be my second choice.

Orange shares a school district with Woodbridge and Bethany. Also great towns. A bit less convenient to the shops and restaurants you listed, but they are quieter.