When he refers to Stephen as the "lowest of the low" he isn't talking about his importance at Candyland. He's talking about him as a person specifically the fact that Stephen does everything he can in order to keep black people enslaved and ensure he has the absolute easiest life he can as a black man in the South during that time period.

He's essentially calling Stephen the biggest piece of shit out of the entire pile of shit.