Jason Brennan, a political philosopher at Georgetown and author of the new book Against Democracy, generally writes from a libertarian perspective. He insists there is no contradiction between believing that voters can be remarkably foolish and irrational and a general belief that a society can operate based on the generally uncoerced decisions of these same foolish and irrational would-be voters.
Brennan lays out the reasons at Bleeding Heart Libertarians, including the sociological reasons why libertarian thinkers might be more inclined to come to such negative conclusions about democracy even though the facts and analysis he brings to bear on the question are not based in libertarianism per se:
it isn't surprising that the new wave of democratic skepticism comes from libertarians like Ilya Somin, Bryan Caplan, or me. When you read most democratic theory, you see that most authors revere politics and democracy, viewing them as in some way sacred or majestic. Libertarians will have none of that. As a result, I think they're able to think more clearly about the nature of democracy. For many on the Left and Right, doing democratic theory is like doing theology. For libertarians, it's just comparative institutional analysis. Libertarians have no inherent emotional draw toward or inherent revulsion to democracy. Asking whether democracy works better than the alternatives has no more emotional resonance than asking whether a hammer works better than a screwdriver for a given purpose…..
as to the question of voters in democracy vs actors in the market [and how libertarians can think the latter can do sensible, non-damaging things while the former might not]: The incentives are radically different.
When I make a market decision, I decide unilaterally. If I order a candy bar, I get a candy bar. If I order an apple, I get an apple. Further, in general, I bear the consequences of my decisions. If I make a bad choice for me, I get punished. If I make a good choice, I get rewarded.
Of course, sometimes the consequences take a long time or are hard to trace. Yes, sometimes there are significant negative externalities. Still, there's a feedback mechanism. However dumb people might be naturally, markets incentivize them to be smarter.
In politics, my decision counts for basically nothing. If I stay home, vote for X, or vote for not-X, the same thing ends up happening. We all bear the consequences of the majority's decision, but no one bears the consequences of her individual decision. If I make a bad choice at the polls, I don't get punished. If I make a good choice, I don't get rewarded.
The feedback mechanism sucks. However dumb people might naturally be, politics incentivizes them to stay that way, or get dumber.
Bryan Caplan, another thinker from the libertarian world who questions the probity and sense of voters who Brennan mentions above, wrote for Reason back in October 2007 on "The 4 Boneheaded Biases of Stupid Voters."
I wrote about Brennan's general perspective on democracy in the context of Donald Trump last year.
Katherine Mangu-Ward wrote a long feature hooked off the basic "your vote counts for nothing" analysis in November 2012. I took an op-ed approach to the idea in "Not Voting and Proud" in 2004.