I think the right person includes many things, including timing. He may have been the perfect guy for you at some point but obviously not the perfect guy for you at this time. I would take this as an indication that the anxiety you mentioned is now interfering with what you want for your life and do something about it. Therapy, medication, try different but healthy coping mechanisms, etc. Whatever you need to not be filled with gut churning regret.