A pilot should be on so you're not leaking gas into the house. It's not a severe hazard, unless it accumulates somewhere (as it might in an oven), but gas stinks (due to an added odorant).