The chart shows 338 vs 86 TJ/kg for fusion vs breeder reactor. That's about 4x, not 100x. The U-235 is essential for the process but not the only source of energy in a fission chain reaction, which can convert U-238 to unstable plutonium.

However, as basically every fission reactor out there is a non-breeder reactor, I can see your point. If we count all the non-fissile parts of the fuel in the density equation (which is a debatable metric), then sure, the fuel density goes way down. But ultimately that isn't too relevant for figuring out the efficiency of the process overall.