His ratings aren't appreciably different from Letterman towards the end. However, Colbert hasn't ever come remotely close to the number of viewers Letterman had during his heyday.

In general, late night talk shows have been on a long, slow decline and aren't all that relevant as part of the national conversation any more.