Yes true – if your luggage is stair-friendly, its not too bad, but if you need an elevator it can be difficult. And again, the entire trip is much easier if you already know where you're going. In both GCT and especially Penn, it's easy to follow a few poorly planned signs and get lost in a labyrinth of hallways if you don't already know your way around.

FWIW: From the main terminal in Grand Central, the S platform can be most easily accessed from the southwest hallway (to find south quickly, it's opposite from the train platforms). If you're coming in from the lower level, the S platform is inconveniently located on the opposite side of the station from the escalator that leads up to the market.

Penn OTOH is just a labyrinthine shitshow. I've somehow developed instincts on where to go myself, but the best advice is to just follow signs for NJTransit.