Study Finds Liberal Pundits More Accurate Than Conservatives

In my recent review of the great new book, Future Babble: Why Expert Predictions Are Next to Worthless, And You Can Do Better, I noted the work of political scientist Phililp Tetlock, who set up a 20-year experiment in which he enrolled nearly 300 experts in politics.

Tetlock then solicited thousands of predictions about the fates of scores of countries and later checked how well they did. Not so well. Tetlock concluded that most of his experts would have been beaten by "a dart-throwing chimpanzee." Tetlock found that the experts wearing rose-tinted glasses "assigned probabilities of 65 percent to rosy scenarios that materialized only 15 percent of the time." Doomsters did even worse: "They assigned probabilities of 70 percent to bleak scenarios that materialized only 12 percent of the time."

Now some students at Hamilton College in New York have conducted a similar study evaluating 26 pundits (mostly prominent denizens of op/ed pages and haunters of Sunday morning political talk shows) for the accuracy of their predictions. Basically their research confirms Tetlock's findings. From the executive summary:

We have discovered a number of implications from our regressions and analysis of the data.  First, we have discovered that six of the analyzed prognosticators are better than a coin flip (with statistical significance.)  Four are worse, and the other 16 are not statistically significant.  A larger sample can provide better evidence addressing the question of if prognosticators on the whole are better than a coin flip.  We understand that being better than a coin flip is not a high bar to set, but it is a serious indictment of prognosticators if they are, on average, no better than a flipped coin.

According to our regression analysis, liberals are better predictors than conservatives—even when taking out the Presidential and Congressional election questions.  Whether this holds true from election season to election season needs further evaluation; liberals may have implicit benefits from Obama winning the 2008 election.  Tentatively, however, we can assert that liberals are better predictors than conservatives.  Additionally, individuals with law degrees were less accurate than those who did not possess law degrees.

A final important implications is that we did not discover that certain types of predictions tended to be more or less accurate than others.  For example, we did not see that economic predictions were more accurate than healthcare predictions.  This suggests that prognosticators on the whole have no unique expertise in any area—even on political predictions, like the Presidential or party primary elections.

The Hamilton College website features a good article that shows the rankings of the policy soothsayers:

The students found that only nine of the prognosticators they studied could predict more accurately than a coin flip. Two were significantly less accurate, and the remaining 14 were not statistically any better or worse than a coin flip.

The top prognosticators – led by New York Times columnist Paul Krugman – scored above five points and were labeled "Good," while those scoring between zero and five were "Bad." Anyone scoring less than zero (which was possible because prognosticators lost points for inaccurate predictions) were put into "The Ugly" category. Syndicated columnist Cal Thomas came up short and scored the lowest of the 26.

Even when the students eliminated political predictions and looked only at predictions for the economy and social issues, they found that liberals still do better than conservatives at prediction. After Krugman, the most accurate pundits were Maureen Dowd of The New York Times, former Pennsylvania Governor Ed Rendell, U.S. Senator Chuck Schumer (D-NY), and former House Speaker Nancy Pelosi – all Democrats and/or liberals. Also landing in the "Good" category, however, were conservative columnists Kathleen Parker and David Brooks, along with Bush Administration Treasury Secretary Hank Paulson. Left-leaning columnist Eugene Robinson of The Washington Post rounded out the "good" list.

Those scoring lowest – "The Ugly" – with negative tallies were conservative columnist Cal Thomas; U.S. Senator Lindsey Graham (R-SC); U.S. Senator Carl Levin (D-MI); U.S. Senator Joe Lieberman, a McCain supporter and Democrat-turned-Independent from Connecticut; and Sam Donaldson of ABC.

Landing between the two extremes – "The Bad" – were Howard Wolfson, communications director for Hillary Clinton's 2008 campaign; former Arkansas Governor Mike Huckabee, a hopeful in the 2008 Republican primary; former House Speaker Newt Gingrich, a Republican; Sen. John Kerry of Massachusetts, the Democratic nominee for president in 2004; liberal columnist Bob Herbert of The New York Times; Andrea Mitchell of NBC; New York Times columnist Tom Friedman; the late David Broder, former columnist for The Washington Post; Chicago Tribune columnist Clarence Page; New York Times columnist Nicholas Kristof; Hillary Clinton; and conservative George Will.

Go here to download the study.

Addendum: Megan McArdle, the business and economics editor for The Atlantic, poohs-poohs the statistical analysis of the study. She makes an insightful comparison between the selected prognosticators and how well mutual fund managers do over time:

…my finance profs taught me that the top mutual funds in a given year are not any more likely to show up as next year's top funds.  Indeed, they may be less likely to do well the next year.  Why?  Because funds have strategies, which do better or worse depending on market conditions.  The funds that do well in a given year are probably the funds that were especially well positioned to show outsized fluctuations in response to whatever changed that year--but that also means that they're especially likely to do lose money when those conditions change.  Because the flucuations are a random walk, they do not vindicate the fund manager's strategy or perspicacity--but they may seem to, temporarily.

Which may cast some light on why liberal pundits did especially well in this test.  If you were the sort of person who is systematically biased towards predicting a bad end for Republicans, and a rosy future for Democrats, then election year 2008 was going to make you look like a genius.  If you were the sort of person who takes a generally dim view of anything Democrats get up to, then your pessimism was probably going to hit more often than it missed.

It would be interesting to go back and look at the same group in the year running up to 2010.  But even then, it would tell us very little.  To do any sort of a true test, we'd have to get a bunch of these prognosticators to all make predictions about the same binary events, over a lengthy period of time, and then see how they fared over a multi-year period. I suspect that they'd end up looking a lot like mutual fund managers: little variation that could be distinguished from random variance.

It would be cheaper and just as accurate to hire one of those dart throwing monkeys, I guess.

Disclosure: While I still read op/ed pages, I have not tuned into the Sunday morning political talk shows for a number of years now. Also, I have a niece who will be graduating from Hamilton this month. She studied geology and archaelogy and did not participate in the research.