There are @ 100 min express trains (fewer stops) from new haven to grand central. My company was walking distance from grand central. Having to jump on another subway from there will also had time and headache. Took about 35 -40 minutes to drive and park in new haven from a town just south of Hartford. All in all it averaged about 2H 45m each way. Driving in is possible, but you are looking at needing to leave very very early.

I now drive to Boston one day a week from the same area and leave around 5. Arrive just before 7. DM me if want more info.