Crowdsourced Amateurs Outperform CIA at Predicting World Events


US Govt

Elaine Rich is a pharmacist in her 60s. She and a team of 3,000 other amateur forecasters in the Good Judgment Project (GJP) use Google to keep current on the news. The Central Intelligence Agency (CIA) employs over 20,000 professionals, operates with an annual budget north of $14 billion, and has access to oodles of classified information.

Which of these groups is better at predicting world affairs?

When it comes to "everything from Venezuelan gas subsidies to North Korean politics," reports National Public Radio (NPR), amateurs outperform the pros. Rich, in particular, has "been put on a special team with other superforecasters whose predictions are reportedly 30 percent better than intelligence officers." NPR explains how this is possible:

"Everyone has been surprised by these outcomes," said Philip Tetlock, one of the three psychologists who came up with the idea for the Good Judgment Project. The other two are Barbara Mellers and Don Moore….

But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people….

"There's a lot of noise, a lot of statistical random variation," Tetlock said. "But it's random variation around a signal, a true signal, and when you add all of the random variation on each side of the true signal together, you get closer to the true signal."

The GJP has been operating for about three years. Tetlock's team provides people like Rich with some basic training in probability estimation, and then they're good to go.

This network of folk forecasters isn't likely to supplant the CIA, but it is looking to make changes in the way the intelligence community operates. The GJP blogged this week that "for many geopolitical forecasting questions, we see promise in a human-machine hybrid approach that combines the best strengths of human judgments and statistical models."

NEXT: Steve Chapman on the Push To Send Troops to Ukraine

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. “for many geopolitical forecasting questions, we see promise in a human-machine hybrid approach

    Yeah the CIA already has that. It’s called hooking your balls up to a car battery.

    1. “we see promise in a human-machine hybrid approach”


      1. I am the future.

        1. Speaking of which, I finally read the Warty Hugeman DC o’ D serial.

          Hilarious stuff. Well written, too. You have a genuine gift, sir. {SPOILER ALERT} I’m still laughing at the “Find My Penis” app.

          1. My fave so far was the hand to penis combat.

          2. Speaking of H&R literary genius, who was it that wrote De Bello Lemures?

            1. who was it that wrote De Bello Lemures?


            2. Fluffy.

              Or Thomas Brookside.

              Neither are his real name, I think.

            3. In a surprise to Fluffy, I preferred Last Days of Jericho.

          3. Thank you very much.

            The final chapter will come out on Monday.

            1. Is this the “Red Bedding” I’ve been hearing about?

              1. Is this the “Red Bedding” I’ve been hearing about?

                Yes, but it plays out differently on the show.

                1. Less killing, more maiming?

                  1. Less killing, more whorexposition.

                    1. Okay, so an orgy with people wearing red cloaks? This is starting to sound like a Kubrick film.

                    2. They all have animal masks on and a few of them have two.

                    3. This should be the scene where you bring back to life all of the characters that you’ve killed off, just to kill them again, even more violently than before.

                    4. No, JW. I am not going to rip-off The Facts of Life series finale.

  2. How is this different than the scam of taking a list of people, telling half of them to go long and half to go short on Stock X, then when X goes down, taking the list you told to go short, tell half to go long and half to go short on Y, rinse, repeat, until you have an eighth of your original list to whom you look like a genius?

    1. What I mean is, they started with a room of 3000 people, it’s no surprise that a few would end up looking like genius forecasters, but what’s to prevent them regressing to the mean eventually?

      1. Some of their results might be related to a lack of bureaucratic ass covering, institutional politics and general civil servant apathy?

        1. In office politics, it doesn’t pay to make noise about predictions that aren’t the boss’ pet prediction. It ever so subtly slows down your career…

          1. Sometimes not even subtlely….

        2. Some of their results might be related to a lack of bureaucratic ass covering, institutional politics and general civil servant apathy?

          Bingo! I immediately assumed that the lack of needing to advance a political agenda would make their predictions more objective, and thus more accurate.

      2. Depends on their methods. Too lazy to RTFA, but how big is this team of “special” forecasters and how many predictions did they make?

        1. *closes eyes, puts fingertips on temples*

          I’m sensing…that I will be the only one…to respond to your query…

      3. My quick read was that the average of all of the predictions by the amateur group was better than the average from the CIA.

      4. I thought the same thing. I expect this “team of superforecasters” to do worse than the rest of the forecasters in the future.

        1. But they haven’t. That is actually then point of the article. It addresses the outlier scenario. The smaller group consistently ends above average.

          Look at it like Intrade or the stock market. Some people are better at it than others.

          1. Also, this was on the front page of Hacker News, which I am now positive that at least two reason staffers have bookmarked. They discussion there covered the math pretty critically.

          2. Look at it like Intrade or the stock market. Some people are better at it than others.

            That is not a good analogy. Efficient markets, dude.

            But maybe you’re right about the forecasters here. I didn’t RTFA. Again, I wonder how the story would change if these people had some skin in the game, i.e. if it were turned into a prediction market.

            1. I see no difference with this and a market. The incentive is simply personal and internal. Now add more incentive? Yeah I bet it would become even more accurate.

            2. Efficient markets?

              Arbitrage exists.

  3. But also, if you take a large crowd of different people with access to different information and pool their predictions, you will be in much better shape than if you rely on a single very smart person, or even a small group of very smart people.

    And considering that the CIA has neither a single smart person or a group of smart people, the handicap can’t be overstated.

  4. This network of folk forecasters isn’t likely to supplant the CIA…

    Nope. That’s because the CIA has never been about precision. If it were, it would have been boarded up a long time ago.

  5. How hard can it be to be more accurate than the fucking CIA that didn’t even predict the fall of the USSR?

    1. The same CIA that misjudged the ability to oust Castro, believed the US would be greeted as liberators in Iraq, and failed to prevent 9/11?

    2. I remember looking at the CIA’s world fact page and seeing that its description of the US economy says that “Economic inequality is a real problem”.

      I am guessing the CIA will not predict the coming collapse of the stock market bubble and the following implosion of the US economy either.

      It is like asking bunch of Marxists to predict when their own fundamental flaws will destroy everything they have built.

      1. Well I would assume there’s quite a few people in the CIA who are well aware of the coming collapse. They just went ahead and bought property on Whore Island, and just working to maintain the payments on their getaway jet.

      2. I always turn to Mali when looking at an atlas. It brightens my day and make me more gracious than I otherwise would be to know that I don’t live there.

    3. CIA vs Ouija board, who wins?

      1. the Afterlife

      2. Fuck ’em both. Use the Magic 8-Ball.

        1. try again later

  6. What exactly have they predicted and how early? Also, praise from David Brooks on your homepage? It makes me skeptical.

  7. Sounds liek some pretty serious business dude.

  8. “Everyone has been surprised by these outcomes.”

    Particularly the CIA.

  9. Clearly we should contract out more intelligence work to ISIS.

  10. I wonder how much better their predictions would be if they had some skin in the game.

  11. seems the most obvious difference is that the amateurs are neither angling for professional advancement nor agitating for spending that supports their predictions.

  12. I don’t see this as a surprise. As a matter of fact, if you replaced “Crowdsourced Amateurs” with “Monkeys With Keyboards” and left the rest of the lede alone, I’d find it equally likely.

    1. Ah, I see you’re familiar with the research done in the Simianspear Project.

    2. I wouldn’t. From an actual study:

      Not only did the monkeys produce nothing but five pages consisting largely of the letter S, the lead male began by “bashing the hell out of” the keyboard with a
      stone, and the monkeys continued by urinating and defecating on it.

      1. OK, fine. So their analysis is exactly like the CIA’s.

      2. That was fun – thanks for that, sarc!

      3. Phillips said that the artist-funded project was primarily
        performance art, and they had learned “an awful lot” from it. He
        concluded that monkeys “are not random generators. They’re more complex
        than that. ? They were quite interested in the screen, and they saw
        that when they typed a letter, something happened. There was a level of
        intention there.

        Why do I suspect a similar set of comments was once made by one of our troll’s high school English teachers?

  13. It was the best of times it was the blurst of times…YOU STUPID MONKEY!

    1. I find myself saying that, a lot. Man, I miss the good Simpsons.

    2. Best. Simpsons. Ever.

  14. To me, this isn’t surprising at all. Check out the book “The Difference” by Scott Page. He’s a stats prof at UMich. Done a ton of research on this subject – pretty much all of which would suggest we should expect the results this group delivers.

    Saw him speak at my workplace – bought the book immediately after. Interesting stuff – highly recommend it.

  15. Let’s not forget that the CIA is not only predicting events, but attempting to influence them as well. They have a built-in bias predicting their influence will work as planned. I suspect simply betting against the CIA’s predictions 100% of the time would also lead to better averages.

  16. Two CIA recruiters paid a visit to my university about a month ago, and their panel was pretty disappointing. One of them seemed really drab, and made the clandestine service sound like a boring desk job. The other guy was really scrawny, and didn’t look at all like an American James Bond. Needless to say, I ain’t looking for a career with the Angency anymore.

    1. Agency*, I’m not trying to get philosophical on you all.

  17. Yeah, but without the CIA, who would do the ‘ssassinatin’?

    1. O! Drones, guided by the “Disposition Matrix”

  18. “a human-machine hybrid approach”

    Come vith me if you vant accurate analysis…

  19. have been trying much the same thing, but taking the bias out of it by using events that quantum theory predicts should be truly random: nuclear disintegrations. If quantum inseparability also holds, then these events are as related to everyone’s psyche (or physical manifestations of same) equally, rather than asking the opinions of human-selected individuals. The problem now is relating the data to historic significance.

  20. This is overall a good idea and one I think the CIA is interested in using – and that is crowd sourcing predictions to the extent it can be.

    As others have noted here, just a prediction game likely wouldn’t be useful enough – a better answer is a futures’ market on geopolitical questions.

    However, during the Bush administration an “unnamed source at the Pentagon” was said that they were working on an idea to build a futures’ market to predict terrorist events.

    The idea went away a short time after – because publicly it wasn’t well received.

    This is due chiefly to the fact that the idiots had an easy (though simplistic) argument with “so Bin Laden could’ve made money by betting on an attack would happen on US soil prior to 9/11?” and the argument for the market was (and still is) a lot tougher to make.

    But in general, the CIA’s wish is to be right and if such a futures’ market existed, they would certainly exploit the information it contained.

    I just don’t think they, nor anyone else knows, a way to build the market with minimal public/loud objections, while still building it in such a way to get a fairly large number of people to actively participate and being able to pose “questions” which are truly useful.

  21. The idea of crowd sourced information being more accurate than a very intelligent single person, or even a very intelligent group of people, is not new. It is called the market.

  22. People forget the CIA’s #1 job. When some foreigner sits down with a congressman, said congressman wants to know in advance exactly what that foreigner wants and how he’s going to ask for it. And he asks the CIA to tell him. The CIA is very, very good at precisely this.

Please to post comments

Comments are closed.