I thought the rule would be if something felt off, always trust your instinct. Especially if all the enemy needs is an "invitation"

You could've tried this, ask them to try to enter without permission. Then, observe their reaction.

Tell them you won't harm them for trying, it's just that you're not giving explicit permission to enter