You mentioned The Millenium building. While that is technically downtown, it is a bit isolated in that you won't walk out your door and be mere steps from other things to do, neighbors, etc. It is a converted hotel and faces a highway entrance ramp and road that get VERY busy on nights when there are baseball games or concerts at the amphitheater. Do a Google Street view "drive" to see what I mean in terms of being isolated from the rest of downtown.

If you are looking for a modern building check out The Pennant (right behind The Millenium but on Main Street so not as isolated), Hartford 21, or anything by Spectra. There are also some older buildings that have been converted to nice apartments: Capewell, 55 On The Park, The Metropolitan.

I have lived in the South Downtown neighborhood (north of Element 250, south of the Traveler's Insurance Tower) for 18 years and it is a great neighborhood with a number of historic condos and apartment buildings, great neighbors of all ages and walks of life, lots of dogs, and a good vibe. We just had our neighborhood holiday happy hour on Tuesday at a local restaurant and it was a great turnout!

If you are expecting the vibrancy of NYC you will be disappointed. We may not have the density of other cities but there are a lot of activities/parks/restaurants within a 1-mile walkable radius of downtown. I have never felt unsafe here, walking around at all times of the day or night. Just like any other city, be aware of your surroundings and be smart, don’t leave stuff visible in the back of your car enticing people to break in, etc.