Sophisticated Econometrics: Does It Change Minds?

Some Monday morning economic epistemology for you, via Russ Roberts at Cafe Hayek:

I think too much of modern empirical economics is the economics-free application of sophisticated statistical techniques that does little to actually advance our understanding of the social world….I also don't think we've made "massive progress" in understanding the social world. We've made massive progress in publishing papers on the social world. But understanding? Not so much. We treat the natural world as if the sophisticated tools of statistics can turn reality into a natural experiment. But the world is usually (always?) too complex for the results to be reliable.

I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus—a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work.

When I asked Ian Ayres that question, someone who advocates increased use of statistical techniques in various aspects of life, his answer was the Levitt and Donahue study of abortion and crime. A strange answer as it is highly controversial and widely dismissed by skeptics.

My example used to be the Monetary History of the United States by Friedman and Schwartz that created a consensus that the Fed and the money supply play a crucial role in inflation and business cycles. But The Monetary History is a collection of facts rather than the use of the fancy techniques so in vogue today. I'm not sure there's anything sophisticated in the empirical work. Is there even a single regression in it? I don't know. But the point is that I'm not saying that facts are irrelevant to our understanding of the world. I'm saying that attempts to use statistical technique to tease out causation in a complex world is incredibly un-credible.

The comments thread on his entry is interesting, and doesn't quite meet his challenge. On a much more layman level, see this 1995 Reason magazine review essay by me on the use and misuse of numbers in journalism and policy.

Comments

[Fluffy]

I'm going to threadjack this economics thread to conclude a discussion we've had here many times about the politics of the AIG bailout.

I have argued several times that it wasn't really AIG who was bailed out, but the firms that bought credit default swaps from AIG. There were firms that made asset-backed securities bets that they insured by buying credit default swaps from AIG, who stood to realize large losses if AIG could not pay off on their losing bets. Had AIG been allowed to fail [as they deserved to fail, based on their balance sheet] these firms would have shown huge losses.

Number one of the list of AIG counterparties? Goldman Sachs.

http://www.bloomberg.com/apps/news?pid=20601087&sid=atM1NoT0lfus

Of course, article after article consistently depicts Goldman as one of the "soundly run" firms, that didn't need a government bailout. This is now exposed as a lie. In effect, Goldman had made substantial bets on AIG's soundness as a counterparty, bets that they deserved to lose. They were saved from losing those bets by government action.

[MaterialMonkee]

"When I asked Ian Ayres that question, someone who advocates increased use of statistical techniques in various aspects of life, his answer was the Levitt and Donahue study of abortion and crime. A strange answer as it is highly controversial and widely dismissed by skeptics."

That's the stuff in the freakonomics book?

No one I've ever met that's read the book disagrees with the abortion crime hypothesis

That really is a great book

[John]

I saw that to Fluffy and sent the link to Hit and Run this morning. Bush, Obama, Paulson and Geittner looted the treasury to save Goldman Sachs. It is one of the worst things that the government has ever done. It just makes me sick. The only silver linning to that very dark cloud is that we may not now have the money to socialize healthcare. Not only is that a good thing in its own right, but also lefties may have to face the fact that they won't get government run healthcare despite having huge majorities in both houses of Congress all because the Obamasiah broke the bank saving Goldman Sachs. The thought of that makes me laugh so hard my head hurts.

[John]

As far as the thread goes, I believe Professor Feinman used the proper term once "cargo cult science". That is all it is. For fifty years now economists and social scientists have been doing their regression analysis and the like and have produced zero results.

[hmm]

That's the stuff in the freakonomics book?

yes.

Economics has lost the social aspect of its science. Evey undergraduate to post graduate economist I know thinks the science has turned into a statistics and calculus masturbation exercise. Blinded by math and incapable of seeing what is happening without math. Unfortunately the business end of education has gone the other way and spent decades massaging the people side of the discussion and forgot the math. Recent events of the Harvard case study educated businessmen failing to see the short comings of models like Black Scholes at least points in that direction.

Just like an engineer can prove an elephant can hang from a cliff by holding onto a dandelion with its trunk. I don't a know a single engineer that would stand under the elephant, but more than a few economists will claim without reserve the numbers don't lie. There's a disconnect between reality and the math.

[hmm]

The other thing that bothers me about economists is how many have gone on to actually produce or profit from their science by applying it? I can name hundreds of successful businessmen and women that have taken risks with their own wealth to prove their theory or idea and not one economist willing to put their money where their mouth is and incorporate the risk into the equation and prove it with their own wealth.

Fucking commentators and not doers. worse than armchair quarterbacks.

[Warren]

John,
Would that be Richard P Feynman you're attempting to cite? His more apt observation

All you need to know about economics is:
1 When prices go up, people buy less.
2 People who sell things set prices to maximize profits.
3 Economists know very little.

Dick does DEMAND KURV

[John]

Yes Warren that is who I am talking about, you smartass. I would add to what Feynman said by saying that most of the work done by economists in the last 100 years have been increasingly ellaborate ways around the three things he mentions. Many times macro economic theory is just elaborate snake oil designed to show how we can in some cases avoid the iron laws of supply and demand and productivity and get something for nothing.

[John]

"The other thing that bothers me about economists is how many have gone on to actually produce or profit from their science by applying it?"

I believe Keynes was a master stock picker. But beyond him, I can't think of any. Of course, Vince Lombardi never played pro-football and he seemed to be pretty wise on the subject. John Locke never held public office yet seemed to be pretty wise about how governments should be run. So I am not so sure there is much connection there.

[Fluffy]

Many times macro economic theory is just elaborate snake oil designed to show how we can in some cases avoid the iron laws of supply and demand and productivity and get something for nothing.

In practice there does seem to be little difference between elaboration and obfuscation in economics, true. As in philosophy, the law, etc. If something strikes you as the equivalent of a Ptolemaic epicycle, it usually is.

[LarryA]

I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus-a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work.

None have, but I can think of several that probably should have. For instance, a couple of years back the CDC surveyed gun control studies and concluded that none of them showed any evidence that "common sense" gun control measures actually prevent violence. This replicates the 1970s data by Kates and others in Gun Control; The Liberal Skeptics Speak Out, and numerous studies in between. Yet the "former opponents of one viewpoint" simply refuse to concede. And after twenty years of listening to the gun control predictions of disaster and seeing them fail again and again, people who live in cities and countries where gun prohibition obviously isn't working continue to support ever-stronger/wierder measures. England has developed a supposedly stab-proof kitchen knife that's supposed to turn back the flood-tide of violence they've experienced since banning firearms.

The war on drugs is another such subject. Many people continue to support it despite overwhelming evidence it is horribly wasteful and isn't working.

It's quite possible that the statistical approach "does little to actually advance our understanding of the social world" because too many of us simply refuse to listen.

[Paul]

his answer was the Levitt and Donahue study of abortion and crime.

Don't get me started on this.

That's the stuff in the freakonomics book?

No one I've ever met that's read the book disagrees with the abortion crime hypothesis

That really is a great book

Well, I guess you and I haven't technically met, but put me in the "disagree" column.

[hmm]

If the most basic premise of economics is the rationale person and that that person will be averse to pain or loss and drawn to wealth or gain, then why have the majority of the people seen as experts not profited from their science?

This is not an example of not doing is a measure of understanding. It's an example of following and applying what you espouse. If Locke was in public office do you think he would follow his concepts, if Lombardi was on the field do you think he would follow his own advice? Economists are in the real world where wealth is sought and loss is avoided, why aren't all the economists competing for the top slots of the fortune 500 or worlds wealthiest?

[Paul]

why aren't all the economists competing for the top slots of the fortune 500 or worlds wealthiest?

Interesting question, but understanding the law of supply and demand doesn't mean you're going to get rich from it. Knowing that if there's less of a thing with an increase in demand that the price will go up, is a far cry from identifying those patterns in products or services currently moving through the economy, and subsequently capitalizing on it.

You could ask the same question: Why don't doctors live forever?

[SugarFree]

In 21 years of studying/working on college campuses I have never seen any counter-evidence for the commonplace: "Those who can't do, teach."

[Fluffy]

Economists are in the real world where wealth is sought and loss is avoided, why aren't all the economists competing for the top slots of the fortune 500 or worlds wealthiest?

Well, to be fair, since we don't actually have a market economy, being a great economist might not really confer much advantage in picking a profitable investment.

For example, consider my threadjack. A solid economist might have surveyed the landscape prior to the AIG bailout and said to himself, "AIG is going down. I will bet against them, and bet against the firms that are doing business with them." That person would have gotten fucked over by the bailout. Because it's not enough to understand the economics, you have to understand the politics too.

If you were a great economist and properly identified the real estate bubble as a bubble, you should have been able to make a lot of money, right? But the problem is that if you bet against the bubble too early, you would have been fucked. And the only way to make that decision correctly was by mastering the politics of the situation, not the economics. You had to know when the Fed would choose to pop the bubble and how long it would take them to do so by moving in increments. "Fed Watch" is ultimately an element of political science, not of economics.

[Paul]

Well, to be fair, since we don't actually have a market economy, being a great economist might not really confer much advantage in picking a profitable investment.

To be even fairer, the laws of economics are still not repealed, despite government interventions into same. Obama can stick his fingers in his ears and shout "lalalalalalala" all he wants, but corporate welfare, bailouts and price controls will have all of the expected effect. Spending two dollars when you only have one will continue to have a very predictable result.

[John]

"Economists are in the real world where wealth is sought and loss is avoided, why aren't all the economists competing for the top slots of the fortune 500 or worlds wealthiest?"

You get rich by selling things not by knowing which way the economy is going. There is nothing economics that will teach you what products will sell and what ones won't. Stock picking involves mostly figureing out which companies have the best products and are in the best position to succeed. No amount of knoweldge of how markets work is going to tell you whether company A's product is going to sell better than company B's.

[hmm]

No because doctors have a biological limit that is understood, all living things have such a limit. Do markets and societies have an understood limit of life?

The argument is not solely based on an understanding of the stock market, but the market as a whole. Since economics deals with all aspects of the market, distribution, production, consumption, everything and the social aspects of these variables they should have a much more keen insight into how to generate wealth through any of the means mentioned or observed. Why aren't economists recognizing the social age of information and the rise of Google, the ipod, and such and profiting from such things by getting in on the ground floor? Why do so many of them sit in think tanks and schools and not move into the private sector? They are nothing more than historians that are good at math.

[Sean W. Malone]

Fluffy - to some degree, I think Peter Schiff is somewhat guilty of this... It's one thing to say, "based on current policy, the US Dollar is on the verge of collapse", and quite another to say "The US Dollar will collapse... tomorrow, so sell all dollar-denominated stocks today."

Plus it's hard to say - in that example - what currency might be the prime beneficiary...

You can be a great economist and still suffer from bad timing.

[Sean W. Malone]

John, i think you miss the point entirely if you think the stock market is just about "selling stuff". It's about speculation and trends.

[Sean W. Malone]

BUT... I might note, the good economist will actually be right about the trends & future events, and their causes - even if he's wrong about the specific day that the event happens.

[MaterialMonkee]

"put me in the "disagree" column."

Is that because you found some contrary statistics or have a moral objection to abortion?

The book did make the point that it was a justification for abortion just an unintended consequence.

Also that the reduction in murder rate wouldn't compensate for the loss of life if you consider a foetus a human being

The other explanations for the drop were pretty much pwned in the book

The other bit about Ceau?escu and abortion seems slightly fitting at the moment considering the age demographics in Iran eh?

I actually couldn't pick up if the authors were left or libertarians it didn't seem to have any overwhelming ideological slant.
might wiki the authors

[Paul]

No because doctors have a biological limit that is understood, all living things have such a limit. Do markets and societies have an understood limit of life?

The argument is not solely based on an understanding of the stock market, but the market as a whole. Since economics deals with all aspects of the market, distribution, production, consumption, everything and the social aspects of these variables they should have a much more keen insight into how to generate wealth through any of the means mentioned or observed.

Negative. Read my statements much more closely. See also John's statement above. You get rich with timing, selling stuff, and providing services people want. Not by understanding "buying low and selling high". I've known about "buying low and selling high" since I was but a wee bairne, but now ask me why I'm not the richest man on the planet? You see how understanding the rules to the game doesn't mean you're going to win the game?

John, i think you miss the point entirely if you think the stock market is just about "selling stuff". It's about speculation and trends.

Sean, I think you're taking John too seriously. Speculation and trend spotting is part of buying and selling. It's psychology. It's knowing when an Eskimo could use a refrigerator.

[Paul]

replace "too seriously" with "too literally". aplogogies to you, John.

[hmm]

You do get rich following economic trends. Short sellers made a fortune calling the housing bubble collapse, real estate investors who increased liquidity and became more conservative during the bubble stand to make a fortune on purchasing devalued property. Stock picking involves predicting future cash flows any way you can. A group of people that claim to have a greater understanding of what has happened in an economy should be better poised to identify technical trends and social trends. But they don't. So why do we let them influence legislation that effects our future? Why do we allow such weight to be placed on their opinion and the opinion of some market analyst or marketing researcher? At leas the latter two prover their worth in the market, the economist sits in an ivory tower and claims immunity behind a regression model and calculus.

I'll take advice from a guy who runs marathons regularly before I take the advice of the 100+ pound over weight doctor when it comes to training for a future marathon. I don't care how much science the doctor can throw at me. Just like I would value the advice of a doctor that runs marathons over both of the previously mentioned.

[P Brooks]

Bush, Obama, Paulson and Geittner looted the treasury to save Goldman Sachs.

I thought everybody knew that already.

I'm not picking on you, John. I thought that was plainly obvious long before now.

[Paul]

Is that because you found some contrary statistics or have a moral objection to abortion?

I'm pro abortion all the way. Pedal to the metal. I have a problem with this statistical analisys, and the wide jumps he makes on his conclusion.

I actually couldn't pick up if the authors were left or libertarians it didn't seem to have any overwhelming ideological slant.

I've never known or even speculated about Levitt's political background. I just think his science is flat wrong on this particular subject. I actually liked most of his book.

When I read that chapter, I just smelled something was wrong, went digging, and found my suspicions substantiated for the very reasons I had surmised. He was making very big leaps.

[Paul]

You do get rich following economic trends. Short sellers made a fortune calling the housing bubble collapse, real estate investors who increased liquidity and became more conservative during the bubble stand to make a fortune on purchasing devalued property.

Hmm, you're glossing over the meat and potatoes of what I'm saying. You don't get rich merely knowing the rules. I know the rules, yet I'm not rich. Understanding timing and seeing trends does not require one be an economist. Studying the science of why prices move the way they do has nothing to do...nothing to do capitalizing on trends. Just because an economist understands how money moves through the economy has little to do with capitalizing on a hot housing market.

To take it from another angle, for instance, we know what will happen if we raise tax rates to 100%. But we don't know what will happen if we raise them from 23% to 23.4%. See?

I'll take advice from a guy who runs marathons regularly before I take the advice of the 100+ pound over weight doctor when it comes to training for a future marathon. I don't care how much science the doctor can throw at me.

You sure about that?

[John]

"John, i think you miss the point entirely if you think the stock market is just about "selling stuff". It's about speculation and trends."

Which is just another way of saying figuring out what is going to sell. Knowing that a company has come up with a product like say Crocs, that every five year old and every adult who acts like a five year old must have and buying that company's stock before the price goes up is how you pick stock. Figuring out that some company has developed a new way to mine coal and is likly to get a big contract mining coal is another thing investors do. None of that has anything to do with knowing how the economy actually works at the macro scale.

[Dogzilla]

I think we've regressed in our understanding of the social world. The Romans understood "In vino veritas". Now most people seem to think that alcohol makes you say crazy things that never crossed your mind when you were sober, and I don't really hate Jews I just need to go to AA.
This is just one example. I'm sure I could think of more if I was waterboarded.

[GILMORE]

89.7% of data is actually wrong

[Paul]

Which is just another way of saying figuring out what is going to sell. Knowing that a company has come up with a product like say Crocs, that every five year old and every adult who acts like a five year old

Uh oh, now that controversy. We're elbowing around near the fanny-pack (functional, or just totally ghey?) territory, here. I also believe that men shouldn't wear sandals. At least not in public.

[Mike]

"I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus-a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work."

the failure of Communism

[Zeb]

"Crocs, that every five year old and every adult who acts like a five year old must have"

Maybe some people just like comfortable shoes, regardless of how stupid they look.

[John]

Zeb,

Apparently they do. I wish Ihad invented them.

[Sean W. Malone]

Mike:

The failure of communism had nothing to do with empirical/statistical work. No one read a study and decided that communism was a failure, communism fails under it's own anti-logical premises. Further, without any kind of empirical positivism, simple deduction would suffice to explain it's collapse as a system. No empirical study is needed to tell me that a system in which the state arbitrarily determines prices is going to result in shortages and misallocation of capital resources and that violently taking from producers to give to the leaches of society is going to produce an incentive structure pointing towards failure.

And in addition, the controversy is (thanks to macro-economics/Keynesian horse shit) actually returning.

[Fluffy]

Sean Malone -

I think Mike is playing with you a little.

Strictly speaking, any observation of facts whatsoever constitutes an "empirical study". Therefore an example of an "empirical study" that established a consensus would be everyone's joint observation of the failure of communism, which established at least one consensus.

[Tom M]

"I continue to ask the question: name an empirical study that uses sophisticated statistical techniques that was so well done, it ended a controversy and created a consensus-a consensus where former opponents of one viewpoint had to concede they were wrong because of the quality of the empirical work."

What about "Moneyball"? It's not an empirical study, but Michael Lewis's book about the Oakland A's and how they value players has had some vindication in the baseball world. Newspapers now routinely write about on-base percentage and announcers covering the game do the same, something that was far less likely before the book was written. And more teams seem to be valuing players the way Oakland does.

[carl]

This question just ain't fair. It's the classic creationist question in disguise: point me to one paper or book that conclusively proves evolution. Something like that doesn't exist, because one paper (no matter how good) does not a theory prove. A fairer question would be whether there exists an important question that a collection of papers using advanced statistical techniques provided insight to a difficult question. I think economists have actually done a good job here in a number of fields, but the two that really stand out are understanding male labor supply (a good review is Pencavel 1986 in the Handbook of Labor Economics) and the economic returns to education (Angrist and Krueger, Imbens and Angrist, etc etc).

[Sean W. Malone]

Yeah, I know Fluffy - but actually, you're hitting on something that I don't entirely understand about these debates... And that is, that "empirical" data is all around us all the time and the Austrian school uses empirical data every day because the axioms that their deductions are based around are, necessarily (and kind of by definition), observed and empirical. But we all know that we're not talking about that... We're talking about econometrics and "tests" combined with statistics.

[John]

"What about "Moneyball"? It's not an empirical study, but Michael Lewis's book about the Oakland A's and how they value players has had some vindication in the baseball world. Newspapers now routinely write about on-base percentage and announcers covering the game do the same, something that was far less likely before the book was written. And more teams seem to be valuing players the way Oakland does."

Two things. First, Moneyball won Oakland exactly zero pennents. Second, the subsequent revelations about steroids have I think put moneyball's validity in serious question. The moneyball philosophy with young hitters was to draft guys who hit for high average and on base percentage and then have the "power come later". After Mitchell report we now know that Oakland was the epicenter of steroid use in baseball. Sort of tells us how the "power came later" doesn't it?

[Tom M]

"Two things. First, Moneyball won Oakland exactly zero pennents. Second, the subsequent revelations about steroids have I think put moneyball's validity in serious question. The moneyball philosophy with young hitters was to draft guys who hit for high average and on base percentage and then have the "power come later". After Mitchell report we now know that Oakland was the epicenter of steroid use in baseball. Sort of tells us how the "power came later" doesn't it?"

Moneyball also applied to pitchers, and you're simplifying the arguments about hitting. Moneyball's central conclusion was that "traditional" baseball metrics -- runs batted in, batting average -- weren't good indicators of a player's value. That still holds true, whether the players were chemically aided or not. Just as wins and losses aren't as good an indicator of a pitcher's value as the ratio of walks to strikeouts and the number of home runs he gives up. I personally think Billy Beane undervalues managers, but that's just an opinion and likely not why they haven't won any pennants yet.

[hmm]

You sure about that?

Yes. When preparing for a marathon I will listen to these people in order of weight (I should have qualified that):

a physician that runs (a sports doc)
a runner
a general practitioner that is out of shape

I never dismissed the doctor and do not dismiss the analogical comparison to economists. I just don't give them the weight they think they deserve or the weight they try to demand. To try and boil the argument down to absolutes is a little silly. The point is that people that do have an understanding and connection to what a subject that people that do not do. The argument of objectivity through removal from the subject is bunk when discussing humans and economics. It's a social science not a physical science.

[hmm]

You would also be hard pressed to find a runner that wouldn't recommended regular checkups, especially during strenuous training. Again the people that do, know, the people that comment, have an idea, but don't know.

[jtuf]

Professors get to publish their work if the results are statistically significant. Every professor starts off by analyzing his date with simple, easy statistical tests. A professor will only reach for his book of complicated statistical techniques if that first round of analysis fails to prove anything. The more complicated the statistical techniques in a research paper, the less confidant I am in it's findings.

[John C. Randolph]

There are lies, damned lies, and econometric models.

When I worked at UBS in the 1990s, they employed some pretty high-salaried quants to figure out what kind of trades to make in the derivatives markets. They were losing about a billion dollars a year at that time, on positions they'd taken about five years earlier and couldn't hedge their way out of.

Econometric predictions suffer from the same problem as central planning. You can NEVER have enough information to second guess what billions of people will decide to do.

-jcr