I've been thinking about this and I think you should keep your outdoor shoes beside the chair you always sit in and bring them with you to go out. Then just be sure to enter back through the door you went out of to get your slippers back!

The important part is for both pairs to be slip on. That way it's not so much trouble. I really recommend against wearing slippers indoors and out - changing footwear on entering the house keeps the floor clean!