> What's stopping AI from deciding what's best for humanity if its infinitely more intelligent than us?

Well so far the only "goal" it has been programmed to follow is human instructions. It does that even when it's uncalled for (car hotwiring suggestions for instance). I can totally see that being a reality in your system where you're allowed to be very stupid in a very smart way using AI.