You should have kept this to yourself and politely decline his suggestion. With you sharing secret, he might not consider you friend anymore if he decides to visit. If you can move into other adress that would be nice precaution measure.