Reason Magazine
- reason.com
- reason.tv
- reason.org
- Donate
- Site Search:
Reason on Facebook
Reason on Twitter
Reason on YouTube
Reason RSSGet Reason E-mail Updates!
Manage your Reason e-mail list subscriptionsadvertisements
Why Experts Are Often Wrong
I can't think of two economists who are doing more interesting work than David Levy and Sandra Peart. Their recent book The Vanity of the Philosopher is incredible and they've got a new paper making the rounds about expertise. Here's a snippet from David Warsh's consistently fascinating webzine, Economic Principals:
Levy and Peart's recent work is directed towards the problem of rendering the expert witness more accountable. Indeed, the subtitle of "Inducing Greater Transparency" is "Towards the Establishment of Ethical Rules for Econometrics." (The relatively recently-arrived profession has no code of ethics, formal or, according to many practitioners, informal.) But the problems they are examining are common to a wide range of professional experts, judges, regulators, educators, referees and watchdogs on which society depends to do its business. "Bias" and "capture" of the sort of which, most recently, the credit-rating agencies stand accused are ubiquitous. How, then, to solve what they call "the enormous problem" of harnessing small groups to serve the interests of the general public?
The minimum goal should be transparency, they say. Other people, including the experts' competitors, ought to be able to figure out how they get their results. But transparency won't be forthcoming in, say, the mortgage-lending business, if rating agencies are able to pick and choose from the estimation procedures they apply to the reams of data they are given by lenders. Naturally rating firms will shade their opinions to please the lenders. Economists call this an "incentive-compatibility" problem.
Read the whole thing, which ties in neatly with questions about the subprime mortgage market and a whole lot more.
I interviewed Levy a few years back about his fantabulous book How the Dismal Science Got Its Name (short answer: because liberal economists in 19th century England sought to replace "natural" hierarchy with democracy).
More on dismal science here.
The Wine Commonsewer|8.27.07 @ 12:39PM|#
I was just perusing the latest issue of Sunset and I noticed
that the new experts claim that the rash of fires in the west is
tied to GLOBAL WARMING! Five years ago the same magazine was
claiming that the rash of fires in the west was tied to A CENTURY
OF FIRE SUPPRESSION EFFORTS.
Hmmmmm.
As Tim Cannon used to say:
An expert is a drip under pressure.
|8.27.07 @ 12:55PM|#
So TWC,
What if they are both right...maybe a century of fire suppression
efforts have created a lot of fuel that burns up when climate
patterns change and we get dryer winters?
The trouble is not with the experts opinions, typically. It is
usually with attempts to sum them up in a sound bite. All the
caveats and nuance that make an expert's opinion that of an expert
gets left behind and leaves the appearance of contradiction for the
layman to chew on.
Devil's always in the details.
|8.27.07 @ 12:56PM|#
See new issue of Business Week, cover story on sub-prime mortgage debacle, "Not So Smart." Hundreds of experts who should have known better, none the less said "This time, it is different."
Thomas Ware|8.27.07 @ 1:13PM|#
Cute Frank and Ernest cartoon in yesterday's local rightwing toilet paper - you can line up all the economists in the world and they'll never reach a conclusion.
|8.27.07 @ 1:52PM|#
"When the only tool you have is a hammer,* every problem
resembles a nail."
Why Most Published Research Findings Are False
John P. A. Ioannidis
"There is increasing concern that most current published research
findings are false. ... Simulations show that for most study
designs and settings, it is more likely for a research claim to be
false than true. Moreover, for many current scientific fields,
claimed research findings may often be simply accurate measures of
the prevailing bias."
*The proper toolkit is: hammer, vise-grips, duct tape.
The Wine Commonsewer|8.27.07 @ 2:19PM|#
NM, it's not whether or not one is right, or both are right, or
the nuance. It's the wholesale dumping of one expert to substitute
another expert with a completely different opinion that is rendered
as gospel.
I'm sure that fire suppression plays some part, in fact, the worst
fires in Southern California history resulted from a government
employee refusing to allow a water dropping helicopter to put out a
tiny little brush fire because it 15 minutes past time to call it a
day. The resulting lack of fire suppression was in the news for
weeks killing many people and causing millions of dollars in
property damage.
On a different but related note, I'm always amused by solemn expert
observations that we are moving into the wildlands in unprecedented
numbers when, in fact, all of civilization was once wildlands.
|8.27.07 @ 2:26PM|#
Wow. That's some deeply analyzed research by Mr Ioannidis --
sure hope it's not false.
I think it's long out of print, but back in the '80s I remember
spending about a half-hour in a bookstore chuckling over a book
called "The Experts Speak", which was a collection of quotations
from hundreds of experts about things that we now realize they were
totally wrong about. i.e., leading physicists saying the atom could
never be split, generals saying the aeroplane would never be
militarily important, etc. Made me realize that the only thing
"experts" know is what has previously occurred in their field of
study. When it came to looking toward the future of their fields,
they were remarkably hapless. Sure wish I had bought it.
|8.27.07 @ 3:08PM|#
I always thought that the more you knew, the more you realized you didn't know (orders of ignorance or some such)...by that logic, anyone who is sure of themselves can certainly be considered to be no expert (though I tend to agree with NM's point about distilling expert's opinions into seemingly certain sound bites)
|8.27.07 @ 3:10PM|#
Aside: "Sure wish I had bought it."
Google "The Experts Speak". Top entry is Amazon.com, with 54 copies
available.
Jed Rothwell|8.27.07 @ 3:33PM|#
The book cited above is Cerf & Navasky "The Experts Speak"
(Pantheon, 1984). It has hundreds of examples of mistaken
pronouncements and predictions made by experts. The introduction
says: "Based on our preliminary findings we can say with some
confidence that the experts are wrong without regard to race,
creed, color, sex, discipline, specialty, country, culture or
century. They are wrong about facts, they are wrong about theories,
they are wrong about dates . . ." This view is widely held, but I
think it is corrosive and incorrect. Perhaps this is true of
"experts" in nebulous subjects such as book reviewing or politics,
but it is not true of most technical disciplines.
Most of the incorrect technical predictions made in this book fall
into two categories. The first is mistakes made by self-appointed
experts who had no relevant qualifications or experience. For
example, Adm. Leahy told President Truman in 1945 that the atomic
bomb "is the biggest fool thing we have ever done . . . the bomb
will never go off, and I speak as an expert in explosives." He may
have been an expert in explosives but he knew nothing about nuclear
physics.
The second set of incorrect technical predictions in this book were
not actually incorrect; they only seem that way to the authors, who
know little about the technical subjects and the people they poke
fun at. For example, Harry Guggenheim is quoted regarding Charles
Lindbergh's attempt to fly the Atlantic in 1927: "That fellow will
never make it. He's doomed." Guggenheim knew a lot about airplanes,
and he was 99% right. Lindbergh almost did not make it. He only
survived because he was a fantastically skilled pilot, and he was
very, very lucky. The authors quote the Scientific American in
1915: "the director of military Aeronautics in France has decided
to discontinue henceforth the purchase of monoplanes, their place
to be filled entirely with biplanes. . . . This decision
practically sound the death knell of the monoplane as a military
investment." The authors comment: "far inferior in speed and
maneuverability to a single winged cousin, the biplane soon became
merely a curiosity." This is incorrect. Biplanes are more
maneuverable than monoplanes, which is why they are still widely
used for aerobatics and crop dusting. They took about 15 years to
be replaced after WWI, which was a long time by the standards of
aviation progress back then.
This book is full of similar technical errors. It reeks of the
know-it-all, supercilious attitude of people who do not bother to
do their homework.
Beware of non-experts who claim that experts are wrong. The worst
example of this in modern scientific history is cold fusion. This
research is conducted by distinguished experts and published in
hundreds of mainstream peer-reviewed journal papers, yet which has
been widely denigrated by people who have not read a single paper,
and who know absolutely nothing about the research.
- Jed Rothwell
Librarian, LENR-CANR.org
|8.27.07 @ 4:10PM|#
Mr Rothwell,
That's the one. While standing in a bookstore flipping through the
pages, it didn't occur to me to question the material. It was good
for a few laughs and I suppose its emperor-has-no-clothes aspect
appealed to me. Thanks for providing a more critical point of
view.
|8.27.07 @ 6:03PM|#
TWC,
NM, it's not whether or not one is right, or both are right, or
the nuance. It's the wholesale dumping of one expert to substitute
another expert with a completely different opinion that is rendered
as gospel.
It is the "completely different" part that I take issue with. Most
disagreements between experts are based on a nuanced view of the
issue... they typically (if they are truly experts) agree on the
basic facts.
Take, for instance, wine experts. Most agree on the basic facts
(this one is a very dry red wine), but disagree about the meaning
of the basic facts (its dryness makes it the best choice with
lamb).
|8.27.07 @ 6:06PM|#
FleM,
That is a good article.
An important source of error, often ignored, is when studies are
over powered... making "statistical significance" a very dubious
measure of importance. This is a particular problem in measures of
behavior and intelligence.
|8.28.07 @ 1:51AM|#
What if they are both right...maybe a century of fire
suppression efforts have created a lot of fuel that burns up when
climate patterns change and we get dryer winters?
Only one problem: Since 1930 the temp has only increase .12 degrees
in the lower 48
http://www.climateaudit.org/wp-content/uploads/2007/08/nasa218.gif
|8.28.07 @ 2:10PM|#
Joshua C,
Quick, what's the difference between "dryer winters" and "temp has
only increase (sig) .12 degrees" ?
Pay attention.
|8.29.07 @ 12:30AM|#
Quick, what's the difference between "dryer winters" and
"temp has only increase (sig) .12 degrees" ?
Hey you are the one trying to make spurious links between anthropic
global warming and average US weather patterns, not me.
|8.29.07 @ 2:45AM|#
joshua corning,
You notice that I was responding to a proposition by TWC, right?
You notice that my response was put in the form of a question,
right?
Spurious links were not made by me.
Jed Rothwell|8.29.07 @ 12:12PM|#
Joshua Corning wrote:
"Only one problem: Since 1930 the temp has only increase .12
degrees in the lower 48"
According to the NOAA, 2006 temperatures in the lower 48 were
"2.1°F (1.2°C) above the 20th Century mean." See:
http://www.ncdc.noaa.gov/oa/climate/research/2006/ann/ann06.html
The Mainichi just published an article (in Japanese) claiming that
the NOAA has concluded that the 2006 temperature elevation was
caused by global warming and not El Nino or some other natural
effect. I am looking for a version of this in English, but anyway,
here it is in Japanese:
http://www.mainichi-msn.co.jp/shakai/tenki/news/20070830k0000m030063000c.html
- Jed Rothwell
Librarian, LENR-CANR.org
Site comments/questions:
Media Inquiries and Reprint Permissions:
(310) 367-6109
Editorial & Production Offices:
3415 S. Sepulveda Blvd.
Suite 400
Los Angeles, CA 90034
(310) 391-2245