I honestly can’t think of a better fit than Milford for ALL that criteria. A big part of this is Yale New Haven. That’s going to be your best concentration of specialist medical care in southern New England.

The town has walkable areas (but isn’t ALL walkable, mostly below route 1, downtown, coastal areas) and train access to NYC. Very laid back vibes. Tons of parks and family activities.

West Hartford I see suggested, would be great, but not 2 hours to NYC.

Fairfield would be another option. Plenty of good doctors around there, smaller hospitals.