As a person with a rare disease, I’d advise trying to focus on areas along the New Haven line. You can take the train to NYC (or New Haven) and see any doctor you want. That’s mostly why I commented, as I’m fairly new to CT and we are also looking to purchase a home (but we are middle aged and childless). I wish I had additional advice!

Your budget would certainly get you into many of the nice towns with good schools, but walkability is going to be harder in those towns due to the larger estates.