Here's exactly what to do. Get three or four friends to come over who are ready to help you, and if you truly don't want the fence where it's at, go talk to the guy whether or not he's friendly and explain to him that a surveyor can prove that the fence isn't on his property or in the right of way or anything like that and that you don't want it where it's at and then work out a deadline with him to fix the problem, or give him a specific hard date of when you're going to tear it down and dispose of it and send him a bill for the work. Give him a bunch of reasonable options but be firm and be ready to follow through. If a surveyor backs up your land claim, there ain't shit he can do in court it's all just going to be social politics after that. Record all your conversations, take pictures and don't leave anything to hearsay. If you truly don't want the fence be fair and be firm. You'll have to realize that this dude probably made a mistake and followed through with a big project while being in the wrong so it will more than likely be that he won't swallow his pride and do the right thing. When people fuck up like this you got to put your foot down especially when it comes to the sovereignty of your property.