Of course it's still an option. You can ask your brother for the keys back, low-key tell him that you've changed your mind or just lie and say you're going to be there. It is your house.

At the end of the day, it sounds like you're just not comfortable with this. What I'm trying to explain is that this is totally fine. You can be unapologetic about this. You can be polite but firm. It doesn't matter if you already said yes, you can change your mind. If you're this close to an airport, you're also close to hotels that this guy can stay at. Full stop.

If you're worried about someone opening the fridge and just eating food, then yeah there's nothing you can do short of not letting this person stay in your house. Otherwise, lock valuables up (or take them with you).