This isn't about lawyers and going to court and getting a judge to waive their gavel around. Call him out on his bullshit but be strategic and calm and calculating. Show him the survey, find local law president about how you can legally tear it down and then also have the social standing to make it clear that there's nothing he can do to stop you from doing that and that he can fix it or he can eat it.