While your basic premise of systemic undercounting of Black and Latino workers is correct, that changed during the 1980s (source: https://www.bls.gov/mlr/1999/12/art1full.pdf). Housework by (predominantly) women has never been counted. And the share of white male workers has declined since the 1950s. As the linked article states, “In contrast to the labor force participation of women, those of men decreased significantly during the 1950–98 period” (largely because of better disability insurance; disabled/seriously injured men had no choice but to work 50+ years ago) .And, again, the population has more than doubled since 1954. Even if the BLS were consistently undercounting people of color, we’d definitely notice if roughly a third of the population were still undercounted in labor force participation.

In any case, changes in how the data were collected/interpreted 40 or 70 years ago still don’t account for the fact that we have roughly 22M more workers today than twenty years ago.

Edit: Updated with partial reason for decline among white male workers