I'm not sure why people would recommend against Charles - of all the N-S streets in that area, foot traffic will probably be highest on Charles, which is what you want from a safety perspective. St Paul and especially Calvert are both fairly residential in that area, so you're going to have less foot traffic once the sun goes down.

For what it's worth, if I'm wandering home from Club Charles late at night, I walk down Charles to Mt Royal to get back to Bolton Hill. I've never felt unsafe on Charles after dark, though it can get lonely on the bridge over the Jones Falls. Maryland Ave can be a bit weird at night, and I've heard too many stories about nighttime muggings to feel comfortable on the Howard St bridge after dark.