Why Experts Are Often Wrong
I can't think of two economists who are doing more interesting work than David Levy and Sandra Peart. Their recent book The Vanity of the Philosopher is incredible and they've got a new paper making the rounds about expertise. Here's a snippet from David Warsh's consistently fascinating webzine, Economic Principals:
Levy and Peart's recent work is directed towards the problem of rendering the expert witness more accountable. Indeed, the subtitle of "Inducing Greater Transparency" is "Towards the Establishment of Ethical Rules for Econometrics." (The relatively recently-arrived profession has no code of ethics, formal or, according to many practitioners, informal.) But the problems they are examining are common to a wide range of professional experts, judges, regulators, educators, referees and watchdogs on which society depends to do its business. "Bias" and "capture" of the sort of which, most recently, the credit-rating agencies stand accused are ubiquitous. How, then, to solve what they call "the enormous problem" of harnessing small groups to serve the interests of the general public?
The minimum goal should be transparency, they say. Other people, including the experts' competitors, ought to be able to figure out how they get their results. But transparency won't be forthcoming in, say, the mortgage-lending business, if rating agencies are able to pick and choose from the estimation procedures they apply to the reams of data they are given by lenders. Naturally rating firms will shade their opinions to please the lenders. Economists call this an "incentive-compatibility" problem.
Read the whole thing, which ties in neatly with questions about the subprime mortgage market and a whole lot more.
I interviewed Levy a few years back about his fantabulous book How the Dismal Science Got Its Name (short answer: because liberal economists in 19th century England sought to replace "natural" hierarchy with democracy).
More on dismal science here.
Editor's Note: As of February 29, 2024, commenting privileges on reason.com posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary period. Subscribe here to preserve your ability to comment. Your Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the digital edition and archives of Reason magazine. We request that comments be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Please
to post comments
Cuz them questions is hard.
This sounds like a Marginal Revolution post...
I was just perusing the latest issue of Sunset and I noticed that the new experts claim that the rash of fires in the west is tied to GLOBAL WARMING! Five years ago the same magazine was claiming that the rash of fires in the west was tied to A CENTURY OF FIRE SUPPRESSION EFFORTS.
Hmmmmm.
As Tim Cannon used to say:
An expert is a drip under pressure.
So TWC,
What if they are both right...maybe a century of fire suppression efforts have created a lot of fuel that burns up when climate patterns change and we get dryer winters?
The trouble is not with the experts opinions, typically. It is usually with attempts to sum them up in a sound bite. All the caveats and nuance that make an expert's opinion that of an expert gets left behind and leaves the appearance of contradiction for the layman to chew on.
Devil's always in the details.
See new issue of Business Week, cover story on sub-prime mortgage debacle, "Not So Smart." Hundreds of experts who should have known better, none the less said "This time, it is different."
Cute Frank and Ernest cartoon in yesterday's local rightwing toilet paper - you can line up all the economists in the world and they'll never reach a conclusion.
"When the only tool you have is a hammer,* every problem resembles a nail."
Why Most Published Research Findings Are False
John P. A. Ioannidis
"There is increasing concern that most current published research findings are false. ... Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias."
*The proper toolkit is: hammer, vise-grips, duct tape.
Amazing....Reason is still pushing the collectivist wikipedia agenda.
NM, it's not whether or not one is right, or both are right, or the nuance. It's the wholesale dumping of one expert to substitute another expert with a completely different opinion that is rendered as gospel.
I'm sure that fire suppression plays some part, in fact, the worst fires in Southern California history resulted from a government employee refusing to allow a water dropping helicopter to put out a tiny little brush fire because it 15 minutes past time to call it a day. The resulting lack of fire suppression was in the news for weeks killing many people and causing millions of dollars in property damage.
On a different but related note, I'm always amused by solemn expert observations that we are moving into the wildlands in unprecedented numbers when, in fact, all of civilization was once wildlands.
Wow. That's some deeply analyzed research by Mr Ioannidis -- sure hope it's not false.
I think it's long out of print, but back in the '80s I remember spending about a half-hour in a bookstore chuckling over a book called "The Experts Speak", which was a collection of quotations from hundreds of experts about things that we now realize they were totally wrong about. i.e., leading physicists saying the atom could never be split, generals saying the aeroplane would never be militarily important, etc. Made me realize that the only thing "experts" know is what has previously occurred in their field of study. When it came to looking toward the future of their fields, they were remarkably hapless. Sure wish I had bought it.
So why should be trust experts on the subject of expert wrongness?
I always thought that the more you knew, the more you realized you didn't know (orders of ignorance or some such)...by that logic, anyone who is sure of themselves can certainly be considered to be no expert (though I tend to agree with NM's point about distilling expert's opinions into seemingly certain sound bites)
Aside: "Sure wish I had bought it."
Google "The Experts Speak". Top entry is Amazon.com, with 54 copies available.
The book cited above is Cerf & Navasky "The Experts Speak" (Pantheon, 1984). It has hundreds of examples of mistaken pronouncements and predictions made by experts. The introduction says: "Based on our preliminary findings we can say with some confidence that the experts are wrong without regard to race, creed, color, sex, discipline, specialty, country, culture or century. They are wrong about facts, they are wrong about theories, they are wrong about dates . . ." This view is widely held, but I think it is corrosive and incorrect. Perhaps this is true of "experts" in nebulous subjects such as book reviewing or politics, but it is not true of most technical disciplines.
Most of the incorrect technical predictions made in this book fall into two categories. The first is mistakes made by self-appointed experts who had no relevant qualifications or experience. For example, Adm. Leahy told President Truman in 1945 that the atomic bomb "is the biggest fool thing we have ever done . . . the bomb will never go off, and I speak as an expert in explosives." He may have been an expert in explosives but he knew nothing about nuclear physics.
The second set of incorrect technical predictions in this book were not actually incorrect; they only seem that way to the authors, who know little about the technical subjects and the people they poke fun at. For example, Harry Guggenheim is quoted regarding Charles Lindbergh's attempt to fly the Atlantic in 1927: "That fellow will never make it. He's doomed." Guggenheim knew a lot about airplanes, and he was 99% right. Lindbergh almost did not make it. He only survived because he was a fantastically skilled pilot, and he was very, very lucky. The authors quote the Scientific American in 1915: "the director of military Aeronautics in France has decided to discontinue henceforth the purchase of monoplanes, their place to be filled entirely with biplanes. . . . This decision practically sound the death knell of the monoplane as a military investment." The authors comment: "far inferior in speed and maneuverability to a single winged cousin, the biplane soon became merely a curiosity." This is incorrect. Biplanes are more maneuverable than monoplanes, which is why they are still widely used for aerobatics and crop dusting. They took about 15 years to be replaced after WWI, which was a long time by the standards of aviation progress back then.
This book is full of similar technical errors. It reeks of the know-it-all, supercilious attitude of people who do not bother to do their homework.
Beware of non-experts who claim that experts are wrong. The worst example of this in modern scientific history is cold fusion. This research is conducted by distinguished experts and published in hundreds of mainstream peer-reviewed journal papers, yet which has been widely denigrated by people who have not read a single paper, and who know absolutely nothing about the research.
- Jed Rothwell
Librarian, LENR-CANR.org
Mr Rothwell,
That's the one. While standing in a bookstore flipping through the pages, it didn't occur to me to question the material. It was good for a few laughs and I suppose its emperor-has-no-clothes aspect appealed to me. Thanks for providing a more critical point of view.
TWC,
NM, it's not whether or not one is right, or both are right, or the nuance. It's the wholesale dumping of one expert to substitute another expert with a completely different opinion that is rendered as gospel.
It is the "completely different" part that I take issue with. Most disagreements between experts are based on a nuanced view of the issue... they typically (if they are truly experts) agree on the basic facts.
Take, for instance, wine experts. Most agree on the basic facts (this one is a very dry red wine), but disagree about the meaning of the basic facts (its dryness makes it the best choice with lamb).
FleM,
That is a good article.
An important source of error, often ignored, is when studies are over powered... making "statistical significance" a very dubious measure of importance. This is a particular problem in measures of behavior and intelligence.
What if they are both right...maybe a century of fire suppression efforts have created a lot of fuel that burns up when climate patterns change and we get dryer winters?
Only one problem: Since 1930 the temp has only increase .12 degrees in the lower 48
http://www.climateaudit.org/wp-content/uploads/2007/08/nasa218.gif
Joshua C,
Quick, what's the difference between "dryer winters" and "temp has only increase (sig) .12 degrees" ?
Pay attention.
Quick, what's the difference between "dryer winters" and "temp has only increase (sig) .12 degrees" ?
Hey you are the one trying to make spurious links between anthropic global warming and average US weather patterns, not me.
joshua corning,
You notice that I was responding to a proposition by TWC, right? You notice that my response was put in the form of a question, right?
Spurious links were not made by me.
Joshua Corning wrote:
"Only one problem: Since 1930 the temp has only increase .12 degrees in the lower 48"
According to the NOAA, 2006 temperatures in the lower 48 were "2.1?F (1.2?C) above the 20th Century mean." See:
http://www.ncdc.noaa.gov/oa/climate/research/2006/ann/ann06.html
The Mainichi just published an article (in Japanese) claiming that the NOAA has concluded that the 2006 temperature elevation was caused by global warming and not El Nino or some other natural effect. I am looking for a version of this in English, but anyway, here it is in Japanese:
http://www.mainichi-msn.co.jp/shakai/tenki/news/20070830k0000m030063000c.html
- Jed Rothwell
Librarian, LENR-CANR.org