The difference in tone also works in-universe, and with the framing device of the stories only being translated by Tolkien (from the Red Book of Westmarch). The Hobbit is Bilbo's story, written down half a century after the events, a period of time in which he has probably bored the whole of the Shire retelling the tale of his adventure, so the willing audience becomes those who haven't heard it yet, young children. The Lord of the Rings is Frodo's story, an account of struggle and war written in the immediate aftermath of the events by someone with PTSD.