Yep, that's right. We're just talking about two different things.

To visit the Senate gallery (located in the Capitol building) you need the gallery pass, so you should be good to do that.

To attend a committee hearing (located in Senate office buildings) as the original commenter suggested is much more difficult.