I do exactly this as well. I wear fleece lined leggings with sweatpants over them and a long sleeve shirt and hoodie inside all winter. I keep the downstairs at 55 and the upstairs at 65 (I have a senior cat so I keep her part of the house warmer. She spends 90% of her day on my bed)