Vermont was generally late to industrialization and never industrialized to the same extent of really any other Northeastern state, even Maine or New Hampshire. Its further from major ports (the Connecticut River isn't easily navigable past Greenfield, Mass and goods going to the St. Lawrence would have to pass an international boundary) and the logistics of freight rail in the mountains for the most part didn't make financial sense until the quarries in Rutland and Barre became highly valued export commodities.

One thing people are leaving out beyond immigration is that similar to Massachusetts 50 years earlier, there was a big migration of people from rural towns in Vermont into larger industrial centers during this period looking for better paying jobs as well as education and other services you couldn't get in farming towns. If you look at the census data, the populations in the small towns in Vermont fell basically across the board in the late 19th century while the bigger towns like Rutland or Barre grew dramatically.