Sophisticated Econometrics: Does It Change Minds?

Some Monday morning economic epistemology for you, via Russ Roberts at Cafe Hayek:

I think too much of modern empirical economics is the economics-free application of sophisticated statistical techniques that does little to actually advance our understanding of the social world….I also don't think we've made "massive progress" in understanding the social world. We've made massive progress in publishing papers on the social world. But understanding? Not so much. We treat the natural world as if the sophisticated tools of statistics can turn reality into a natural experiment. But the world is usually (always?) too complex for the results to be reliable.

I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus—a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work.

When I asked Ian Ayres that question, someone who advocates increased use of statistical techniques in various aspects of life, his answer was the Levitt and Donahue study of abortion and crime. A strange answer as it is highly controversial and widely dismissed by skeptics.

My example used to be the Monetary History of the United States by Friedman and Schwartz that created a consensus that the Fed and the money supply play a crucial role in inflation and business cycles. But The Monetary History is a collection of facts rather than the use of the fancy techniques so in vogue today. I'm not sure there's anything sophisticated in the empirical work. Is there even a single regression in it? I don't know. But the point is that I'm not saying that facts are irrelevant to our understanding of the world. I'm saying that attempts to use statistical technique to tease out causation in a complex world is incredibly un-credible.

The comments thread on his entry is interesting, and doesn't quite meet his challenge. On a much more layman level, see this 1995 Reason magazine review essay by me on the use and misuse of numbers in journalism and policy.