Either the friend is racist or implying that there are racists in Vt. Both are probably true. I'm white, so can't attest to much, the racism I've seen here is well-meaning white people being racist. I'm sure there are horror stories of much worse (and there are plenty of news articles confirming the more overt racism, see Kiah Morris for example).

Vermont is a very white state. Once outside the greater Burlington area, racial diversity nosedives. If you are people of color, consider whether you'd want to be one of the few families of color in whatever Vermont town you choose. Most folks would welcome you with open arms (though slowly, as they do with everyone), but they're might be a few that really suck.