Maryland was a plantation state, and with that came all the aspects of an agrarian way of life including weak/few cities and towns. Baltimore's rise as an industrial city countered that mostly in the 19th century, but all the area around the city was sparsely populated or developed until post WWII. As with many Eastern cities Baltimore's infrastructure was usurped for use in building the suburban sprawl around it. Add to this until the 1950s Baltimore was the older wiser sister to Washington DC, sharing many things, but after the 2nd war Baltimore became the old maid while DC the jet set younger sibling. Most people in the DC area have never been to Baltimore. Baltimore has many firsts, in the country, and also is known for its quirkyness. Possibly due to its earlier industrial days and the mix of immigrants with Appalachian folks.

I lived in Philly for a short while and had to venture into NJ for work. I think you as a native of NJ, will find Baltimore very comfortable.