It's not just the weather that's subpar in April, but the scenery too. Just a bunch of bare sticks and lots (and lots) of mud, if there isn't snow. The mud can get pretty bad, so much that cars can get stuck on dirt roads, so I can't imagine what that'd be like on a bike. There are also guidelines in place regarding hiking during mud season to protect the trails and vegetation.

The trip you're planning sounds lovely, really, but as others have said, if Vermont is the goal, it's better suited for a different month.