I went this past weekend, the NE regional is easy and gets you there in 2 hours. I think there's always a late NE regional (like 11:30) out of 30th street to DC on Sunday nights.

As far as being an opposing fan, you should be fine as long as you're not a dickhead. You'll get booed at and jeered but you're not going to be in physical danger.