To add to the other comment I'd write up a little contract. Doesn't need to be long or professional just something that says x will pay y and if z happens x gets y back etc.

No reason to start a conflict with neighbors over something trivial because of a misunderstanding.