The drugs/crime level in VT are super low compared to FL, even in the worst spots. It's all relative.
Figure out if you can be ok with being away from city things like grocery stores, takeout, movies, etc. I'm 45 min from the nearest stoplight and get my groceries once a month. I live in a pretty rural area (obviously). If you need those city trappings, you'll have to live in a larger town. Best of luck with the housing situation.
Btw, I grew up in VT but moved to FL in my early 20s. I escaped 16 years later and moved back home. Best decision ever.