I've lived all over New Haven. Maybe we have a different definition of safe. As a general rule none of those people are out to hurt you. Most of them won't bother you at all if you don't bother them. Keep your head about you and treat everyone you come across as a human being and you'll generally be fine.

Also, there's enough areas of New Haven that don't have any of what you mentioned.