Expert Predictions Often Aren't

Ronald Bailey | January 2, 2006

Just fresh off of my own stab at making predictions, I note that Louis Menand has a fascinating review in The New Yorker of University of California-Berkeley psychologist Philip Tetlock's new book, Expert Political Judgment: How Good Is It? How Can We Know?. Tetlock signed up 284 experts and asked them to make predictions about various events over a 20 year period. According to Menand, Tetlock found,

"that people who make prediction their business--people who appear as experts on television, get quoted in newspaper articles, advise governments and businesses, and participate in punditry roundtable--are no better than the rest of us...Tetlock claims that the better known and more frequently quoted they are, the less reliable their guesses about the future are likely to be. The accuracy of an expert's predictions actually has an inverse relationship to his or her self-confidence, renown, and, beyond a certain point, depth of knowledge. People who follow current events by reading the papers and newsmagazines regularly can guess what is likely to happen about as accurately as the specialists whom the papers quote. Our system of expertise is completely inside out: it rewards bad judgments over good ones."

Whole thing here.

Comments

|1.2.06 @ 12:07PM|#

Ron, what's your analysis of Ray Kurzweil? He is extremely radical, yet I think he makes some good points.

|1.2.06 @ 12:10PM|#

I suppose this is pretty easily explained by the fact that more radical predictions are more likely to get attention...and radical predictions are of course more likely to be wrong. That takes care of most it.

Now, where are my flying cars?

|1.2.06 @ 12:24PM|#

If anyone could predict the future, even a little bit, they would be rich. Very rich. Never trust predictions from someone with less than a billion dollars.

Screw the flying cars, I wanted jet-packs.

|1.2.06 @ 12:24PM|#

I like what mtc said.

It may also explain why the insightful folks at Reason get less attention than Ann Coulter and Al Franken.

James Anderson Merritt|1.2.06 @ 12:26PM|#

mtc asks, "Now, where are my flying cars?"

Try www.moller.com. Nieman-Marcus appears to be selling the Moller prototype for $3.5M.

MLER stock would seem a argain at the moment: well under $1 per share, at last trade on Friday.

As far as the "expert opinion" scam, I concluded much the same about the TV weather-guessers a long time ago. Just show me the local and regional dopplers and the hemispheric enhanced satellite loop, and I'll guess the weather for the coming week as well as the onscreen "Experts" or their computer models. In most other disciplines -- politics, the economy, technological development, etc. -- you must similarly be pretty high up to be able to see the big picture well enough to prognosticate with any accuracy. Unfortunately, most pundits are not high up, only high.

|1.2.06 @ 12:35PM|#

So I guess this means we can stop reading Reason?

Screw the flying cars, I wanted jet-packs.

I hear mediageek's backback's got jets...

|1.2.06 @ 12:36PM|#

Uh, that would be backpack.

|1.2.06 @ 1:13PM|#

I, for one, welcome our dart throwing monkey overlords.

|1.2.06 @ 1:19PM|#

Professor Tetlock is working on several other research projects:

(1) "The Pope: Does he wear a funny hat?"

(2) "Where do bears go to the bathroom?"

(3) "Does Mike Tyson have a volatile temper?"

(4) "Are most sit-coms kind of stupid and predictable?"

|1.2.06 @ 1:22PM|#

This is especially true of financial industry prophets. I recently looked up some "top 10 stocks to own for 2003" lists from a bunch of money magazines and TV shows - they almost universally had negative returns.

|1.2.06 @ 2:24PM|#

In reading through some of the links in the article, I came across this hilarious gem regarding the war on drugs and the low price of recreational drugs:

"Mephamphetamines, which can be produced cheaply, have become a growing concern. Their use is growing, especially in the male gay community, where they serve as a lubricant for sexual escapades. This use has had the added effect of increasing the HIV infection rate among gay males."

Holy Santorum! These guys are doing it all wrong! Meth must be pretty damn cheap if it's being adopted as a substitute for Astroglide.

I guess it's pretty easy to confuse "stimulant" and "lubricant" and jump to outlandish conclusions if you're an imbecile.

Mike Laursen|1.2.06 @ 2:34PM|#

The most egregious predictors of the future are the experts that predict ridership right before the taxpayers are asked to increase their contributions to local public transportation systems. We're going through it again in the south end of the San Francisco Bay, after the last sales tax increase we approved for the purposes of extending the BART system ended up being spent to maintain the current system.

|1.2.06 @ 2:48PM|#

In Seattle we have Gary Nickles, who calculated the number of new jobs that will be created by his pet South Lake Union project by adding up the number of buildings to be built, multiplying by the number of cubicles that could be crammed into them, and predicting that's how many jobs are going to be created. At least, that's how I think he came up with his numbers and since I'm not a professional I must be right.

|1.2.06 @ 5:28PM|#

A very funny article. I especially liked the part about the rat outsmarting the Yalies simply because it wasn't ashamed to be wrong forty percent of the time.

This is actually very useful research. We would all like to be categorized at "foxes" but we are nearly all "hedgehogs." We know quite a bit about one or two things and assume what we have learned about them can be extrapolated into every field, "spreading our imperfect knowledge like a male lion spreads his semen," to use the words of Donald Westlake.

But, more than anything, it puts the lie to the notion that we should trust the government because they are "experts with inside information." Plenty of people predicted fiasco in Iraq and a couple of people I know, not mere contrarians but persons who really scoured the internet and read all the details of the press releases and reports made public, coolly predicted that no usable Sarin or centrifuges would be found in Iraq. They were not experts but I would certainly class them as "foxes," based on lifetime performance.

|1.2.06 @ 5:30PM|#

WOOOOOOOOOOOOOOOO-HOOOOOOOOOOO! DOW 10,000!!!!!!!!

|1.2.06 @ 5:44PM|#

As Will Rogers put it, "The weatherman says it's going to rain today, and, you know, he has as much right to an opinion as anyone."

|1.2.06 @ 6:30PM|#

Johnny Clarke-Actually, a heavenly messenger named Xenu appeared before Nickels and gave him the proper numbers.

|1.2.06 @ 7:49PM|#

Not particularly shocking..sometimes I think the "experts" and pundits are TOO close to their particular "area of expertise" - maybe that clouds their objectivity? Interesting..

|1.2.06 @ 8:06PM|#

"As Will Rogers put it, "The weatherman says it's going to rain today, and, you know, he has as much right to an opinion as anyone."

I think the comparison with weathermen is apt to a point. The weather is a chaotic system which measn that exact predictions are very hard to make. But political and social exprerts have the added difficulty that the system they are dealing with is actively trying to to fvk them.

Science and Engineering tend to be exact sciences, because much of what they are dealing with isn't chaotic like the weather. Or if it is, it's the aggregate response of the system that is important.

Being a Weatherman is hard because the public doesn't care about the agregate, they want detailed predictions, yet it simple isnt possible to collect enough good information and process it, to get accurate predictions more than a few days out.

Being a pubic expert is hard because you are deaply involved in the system your making predictions about. You make a prediction, it responds to that prediction. Further unlike Engineering, Science, and the Weather, there is no closing of the loop, where predictions are checked for accuracy, and adjustments are made. (As in Ralph yet fired! This model sucks, change it, this theory falls on it face, etc)

thehim|1.2.06 @ 9:13PM|#

Take the poll here, tell the crazy drug warriors what you think

|1.2.06 @ 9:49PM|#

dead elvis: Nice MC Chris joke there.

|1.2.06 @ 9:56PM|#

Seriously - who here wouldn't kill to be an acknowledged pubic expert?

|1.2.06 @ 10:47PM|#

What is so remarkable about the findings of this book? Pundits make predictions to make news, not to actually forecast the future. The "value" of a prediction is the amount of excitement it provokes, not its accuracy. Listening to a bunch old white men predicting that the economy will grow at a moderate rate, or that Iraqi oil production will rise marginally in the next year wouldn't beat out reruns of "Caroline in the City."

|1.2.06 @ 10:54PM|#

This is where future developments in information markets become our salvation. The stock ticker is pervasive on all news, how much difference will it make when Ann Coulter speaks and a negative growth stock of her cumulative predictive portfolio runs accross teh screen mid sentence. If these folks were really any good at prognosticating they'd be too busy becoming filthy rich doing just that, rather than wasting all that time galavanting about on the talkshow circuit.

|1.2.06 @ 11:13PM|#

> people who appear as experts on
> television, get quoted in newspaper
> articles, advise governments and
> businesses, and participate in
> punditry roundtable...

Um, why would we think people want more from such people than they ask of Bob Denver and Jennifer Anniston? When people tune into bullshit tunes on the radio, aren't they listening for something simple and stupid so they can have the satisfaction of saying "Shit, *I* coulda done THAT one!" Why would this not be so with prognostication?

Fame and excellence correlate weakly... And never more than five times a month.

|1.3.06 @ 1:46AM|#

Predictions of the future? Most experts don't get the present right. I trade commodities and stock indices and am amazed at the BS "experts" spout about what happened in market on a given day let alone what will happen in the future.

One problem is that many of these experts regularly get a certain amount of air time or print space to fill. If they don't understand what is going on they are not going to say, "I don't know why that happened today and I have no idea what will happen tomorrow.", fold their hands and run out the clock.

grylliade|1.3.06 @ 6:36AM|#

Now, where are my flying cars?

<boring science explanation>See, it takes too much energy to hold up something the size of a car to make it worthwhile. The gains in speed aren't worth the loss of economy for short-range transportation, or for bulk, non-luxury goods. Plus, think of the traffic problems, not to mention the idiot factor. Do you really want most people to be flying something the size of a car around at 150-200 mph?</boring science explanation>

As Will Rogers put it, "The weatherman says it's going to rain today, and, you know, he has as much right to an opinion as anyone."

IIRC, meteorology actually has a pretty high rate of correct predictions. The thing is, they give you a "20 % chance of rain," which means that if it rains on one out of five days they predict that, the predictions should be counted as correct. Plus, since they're forecasting for an area, a random observer might happen to rarely be in an area where the prediction is correct. Add in the very human tendency to remember when the weatherman gets it wrong but forget the ten other times when he was right, and you get the idea that weather prediction is pretty bad, when in fact it's pretty damn good.

|1.3.06 @ 6:48AM|#

Hedging by producing arbitrary percentages isn't "pretty damn good." It's barely acceptable. I realize expecting anything approaching 60-70% accuracy outside of Southern California is more or less a forlorn hope, but that doesn't mean I have to praise the guy who waffles worse than a politician in order to hide the fact that his predictions for any later than 48 hours from now are little better than guesses, do I?

|1.3.06 @ 10:24AM|#

From April 1, 2005 Wall Street Journal--

Goldman Sachs Group predicts super spike in oil prices to $105 per barrel.

I keep this in my desk. These people are supposedly putting real money behind these predictions, too.

|1.3.06 @ 12:33PM|#

This is unsurprising... anyone vaguely familiar with Paul Krugman knows this phenomenon already.