Get a quote for a fence that you would like, and tell them that they would need to pay the quote at the end of the construction. Get them to sign something or just like an email, something in writing. Or have them prepay before you consent to having it removed.

Also ask them for a timeline so you can schedule the work to have it restored. Worst case you pay for some part of a new fence, a lot cheaper than lawyers and a lot better to not have neighbors that hate you.