Sounds like they are impacting the use of OP's FULLY OCCUPIED residential property.

People visiting a few times a year do not make a town, the locals do. If it were not for the locals, there would be no town, no amenities, no reason or way for people to own vacation homes.

Locals make an area the way it is. Wealthy folks from outside want to buy a piece of it. They tend to bring the shitty culture they were trying to get away from. If they buy too much, they displace the locals, changing the area and disintegrating what made it charming in the first place.

I'm sorry but I do not see 2nd home owners as having equal rights to people to live in an area full time.

edit: I had more to say...