I was the same way! When I first heard the phrase, I envisioned it as being a reference to being stuck in a deep ball pit. Fortunately I never though the expression was good enough to repeat myself!

Then again, I also thought the "grandfather clause" metaphor was "oh, grandpa's too old to adapt to the new ways, we'll let him keep using the old system", rather than "you can only vote if your white grandpa could".