You own the fence. Dig it up and rebuild it on your property line. Paint it. Do whatever you want, and if the neighbor has an issue, now the cops and courts will enjoy getting involved. As long as they can make money.

Set the fence on fire...do whatever you want fam, it's yours.