They're not wrong that it is a frequently photographed location. It's also historically notable because a former owner has their own Wikipedia article. This doesn't sound like someone walking into the backyard.

The house I grew up in was the only one on a certain stretch of road, then it got subdivided and a new house was built. The new neighbors (not even the original new neighbors, or the second new neighbors) got all NIMBY and filed a noise complaint because we were having a bonfire, in the backyard, with a fire permit. Police determined that the noise level didn't justify a complaint. Taking a photo from a public vantage point where people have done so for years? Did they get misled by the real estate agent? Or did they fail to recognize the property's prominence among tourists and photographers otherwise? Taking photographs from a public vantage point of a picturesque location isn't illegal.

There was also a travel magazine article on this farm's history noting how frequently photographed it is last month: https://newengland.com/today/travel/vermont/the-history-of-sleepy-hollow-farm/. This is a case of rich people expecting the world to revolve around them.

Imagine if that guy in Westford didn't expect people to stop and take photos of that illuminated bird sculpture.