There are several arguments against.

1. Why should the schools have to accommodate a non-student? Especially with limited budgets.
2. Why should a non-student get to take up a spot on a team that a regular student may want?
3. This opens up schools to athletic recruiting shenanigans. If you can get a non-regular student on your sportsball team, there is a significant risk (as we saw in the past leading to the current rule set) that children will be recruited to specific teams.
4. Homeschoolers opt out of the school system for a reason. Getting to pick and chose what parts you want should not be an option. If you want your kid in a public schools athletic program (or art or music) then enroll them public school.

As a child I was home schooled for several years. It was not a good thing and I suffered from it. But god said to do so and thus I was stuck. And while over the decades I've interacted with a small number of children who were helped by it, the vast majority I've met were not.