The Supreme Court and the Wisdom of Crowds

In our October 2010 issue I interviewed Josh Blackman, the founder of FantasySCOTUS.net, a thriving online fantasy league that asks players to predict the outcome of Supreme Court cases. In addition to running the site, Blackman has also been keeping careful track of his fantasy league's results. In a new co-authored paper that's just been been posted to the Social Science Research Network, he reports on what FantasySCOTUS can tell us about crowdsourcing and the Supreme Court. Here's an excerpt from the abstract:

During the October 2009 Supreme Court term, the 5,000 members made over 11,000 predictions for all 81 cases decided. Based on this data, FantasySCOTUS accurately predicted a majority of the cases, and the top-ranked experts predicted over 75% of the cases correctly. With this combined knowledge, we can now have a method to determine with a degree of certainty how the Justices will decide cases before they do. This essay explores the wisdom of the crowds in this prediction market and assesses the accuracy of FantasySCOTUS' predictions about the same cases.

Download the article here.

Comments

[prolefeed]

Based on this data, FantasySCOTUS accurately predicted a majority of the cases

A monkey throwing darts at a dartboard labeled "yes" or "no" has a 50% chance of predicting a "majority of the cases".

"Accurately" doesn't mean what you think it does.

[prolefeed]

Also, I bet most readers of this site could probably beat the FantasySCOTUS crowd with its "accuracy" of somewhat more than 50.0% but less than 75%.

Averaging a crowd's responses means you get a few people who know what they're doing, and a bunch of ignoramuses. Hence, the liberal disappointment with the Mr. Obama they elected.

[dfd]

That was essentially my first though on reading this as well (and admittedly without reading more). With 5000 people making 11,000 predictions it's not at all surprising that some people hit 75%.

[JD the elder]

It's a pretty well-known scam/fallacy, actually; some people take advantage of it to defraud people, while others just fall into it accidentally, which is probably the case with these guys.

Here's how it works: Send out 1000 letters, 500 of which say "Stock A will go up next week" and 500 of which say "Stock A will go down next week". Obviously, half will be wrong; forget about those letters and their recipients. Then send to the recipients of the first, correct letter, 250 letters saying "Stock B will go up next week" and 250 saying "Stock B will go down next week". Again, half will be wrong...rinse and repeat with letters for Stocks C, D, E, etc. After several rounds, you have a series of letters with weeks of perfect stock picks (and, hopefully, a mark who believes you have incredible investment advice). These guys sound like they're doing the same thing, only instead of 1 guy writing 1000 letters, it's 1000 guys writing one letter each.

[Fist of Etiquette]

They made the predictions before arguments were heard?

[Insider traders]

wow...before? that aint free market

[?]

I'm amazed those prediction rates are so shitty. The predictors must be working off the other definition of "fantasy."

Six of the Justices are TEAM! tools, Thomas is a reliable decision-generating machine, Scalia only breaks with TEAM! or Thomas when there's some cop dick they're not sucking right, and Kennedy, when decisive (which he almost never is), goes TEAM!

The SC's output runs over 75% "duh," and the rest is "Oh yeah?duh." If you've ever been surprised by a SC decision, you have your TEAM!s mixed up.

They made the predictions before arguments were heard?

Nothing happens at argument. The cases are made in the brief, and all the Justices' decisions are in there waiting for them?except Thomas's, sometimes, but he's coherent, so you can figure his out on your own.

[Aresen]

Can't look at the data right now. Did the site allow the Fantasy Supreme Court members to bet on how individual justices would vote?

That could prove more interesting.

[Hugh Akston]

More to the point, does it allow you to swap justices to compete in a SCOTUS fantasy league?

[Pro Libertate]

Is there a line on cases in Vegas?

[Otto]

I got Alito and Souter in the draft, so I pretty much stopped adjusting my team prior to decision day...

[JD the elder]

My heuristic for predicting Supreme Court verdicts:

1) What verdict would increase or preserve the power of the Supreme Court? Choose that one.

2) If #1 is a wash, what verdict would increase or preserve the power of government in general? Choose that one.

3) If #2 is also a wash, what verdict would upset the fewest applecarts?

I pretty well guarantee you that'll get you 75% or better right there...

[Troy]

The words "Supreme Court" and "wisdom" in the same sentence cause me a great deal of cognitive dissonance.

[Almanian]

What an utterly useless piece of drivel. I'll never get this minute back...

[Monster Energy Hats]

The words "Supreme Court" and "wisdom" in the same sentence cause me a great deal of cognitive dissonance.

[Links of London]

What verdict would increase or preserve the power of the Supreme Court? Choose that one.

[asics magasin]

nice!