One of the biggest challenges isn't the technology, but getting it into space. Even though there have only been a handful of launch failures, by percentage it is the most dangerous form of travel (besides shark surfing, of course). Putting enough fissile material on a rocket and launching it into space has a pretty high chance of dispersion. Granted, the actual danger of that dispersion, particularly done from an isolated location, is lower than most people would expect, but public fears concerning nuclear energy, and especially critical mass nuclear, creates a political barrier that will likely never be overcome.

If we were ever to get nuclear-powered spaceships, the material will have to be mined and refined off-planet, and there are a lot of steps before we get close to that.