So, I agree with the first part of your statement (term limits are bad), but your rationale is weak.

First, Congress has defunded its own staff capacity since the Gingrich era, so the ability of Congressional staffers to become a "shadow government" is limited.

Second, you actually want a set of staffers with deep expertise in what they do. Like, that's good for government; experienced people who know how to move bills through committee can greatly enhance a young congressperson's ability to pass legislation. Congress' own decision to weaken this is a massive problem, and it's part of why more and more power has concentrated in the hands of party leadership in Congress, because they have the best staff.

Third, you do not have to worry about a shadow government. The US has two branches full capable of usurping Congressional power all on their own; the President and the Supreme Court. And that's part of what's actually occurred. As Congress gets weaker and less capable of making decisions, more and more policymaking is made by the President (via executive order) or the Court (via decisions). This is a really bad outcome. The Constitution only provides for one deliberative, decision-making body: the United States Congress.

Term limits stand to worsen all the current effects, as they generally have in states that have adopted them. The fear here should not be that some anonymous staffers are running the government from within Congress, it's that the two least-representative and least-transparent branches of government will be setting policy.