It seems you ignored your inner voice when you gave him your number. Always listen to that voice.

Tell him the truth! You are uncomfortable communicating with him and don't wish to continue. Respectful goodbye and good luck, then block.