I was in a similar situation when I purchased my first home a few years ago. The most important metrics I considered was cost per BTU, and cost of hardware, installation and maintenance over expected lifetime of the unit. You can easily calculate the cost of electricity, fuel oil, propane, wood pellets, or cord wood per BTU. Then figure in your equipment costs and can pretty easily arrive at a cost leader. For my situation, heating with wood pellets was far more economical over any other solution due to being a cost leader in BTU/$, has one of the longest lifespan, and the second lowest upfront hardware and installation cost in my situation. I was not interested in a traditional woodstove, but I believe that would beat them all.

As far as your comment about electric heating, that is by far the most expensive cost per BTU, especially if you are using electric resistive heating, like a traditional space heater. Remember that heat pumps are not electric resistive, instead of using electricity to make heat, the use electricity to move existing heat. AFAIK they move and “produce” about 3 watts of heat energy for every watt they consume, meaning they are 3x as efficient as a tradition space heater. Everyone’s situation is very different, however I can guarantee that heating with space heaters is by far the most expensive thing you can do.