When you vote, you are exercising ultimate political authority, even if only a tiny fraction of it and even if limited by who is on the ballot (officials) and who isn't ("civil servants", gee thanks, Pendleton Act). With that authority, like all authority, comes the responsibility to use it wisely. Authority and responsibility are converses of one another, and are as inextricable as the positive and negative potentials in an electrical circuit. As dangerous as it is to wield authority without being responsible for the consequences of the orders you give, in equal measure it is absurd to be held responsible for something over which you have no authority to control.

When you slough off the responsibility to use authority wisely ("what if voters are mislead?"), whomever that responsibility defaults-to, takes authority over you ("because the voters could be mislead, we shouldn't let the vote happen!").

When you think about how dumb voters can be, you have a choice how to react. Maybe we should separate into smaller polities so that whoever is dumb will vote dumb things for themselves without affecting the not-so-dumb polity. Maybe we should have been more careful about only letting people vote if they actually have a stake in the system - after all, voting was not a right explicitly granted to anyone in particular, let alone "every citizen", until the 17th Amendment. Or maybe we should just let ourselves be ruled by whomever can best control the popular epistemology in their favor, and hope that their interests are aligned with ours - what could possibly go wrong?