You are not in the wrong. Not at all. He's a grown ass man. He needs to step up and act like one. Just move forward legally. And carefully. Some of these bastards get violent. Pretty sure I simply put the fear of God into the loser. Stand your ground.