Reason Magazine
- reason.com
- reason.tv
- reason.org
- Donate
- Site Search:
Reason on Facebook
Reason on Twitter
Reason on YouTube
Reason RSSGet Reason E-mail Updates!
Manage your Reason e-mail list subscriptionsadvertisements
Science: Mostly A Pack Of Lies
....says scientist John Ioannidis. The Wall Street Journal summarizes his findings:
In a series of influential analytical reports, [Ioannidis] has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
.......
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.
|9.17.07 @ 5:06PM|#
Global warming is probably one of those areas that has (or at least has had) so many scientists scrutinizing the problem, and one another's work, so closely that the consensus is not far off the mark.
|9.17.07 @ 5:12PM|#
Global warming is cooling was
probably one of those areas that has (or at least has
had) had so many scientists scrutinizing
the problem, and one another's work, so closely that the consensus
is seemed at the time not far off the
mark to the majority of the government-funded scientists
jumping on the bandwagon that promised additional funding for
climate research.
Fixed it.
thoreau|9.17.07 @ 5:12PM|#
If this is the guy that I'm thinking of, he has some sobering
observations on the fact that if you try enough variables
eventually you'll find one that, by chance, is statistically
significant.
You may be thinking "What? I thought statistical significance is
all about something not being about chance!" Actually, statistical
significance means that it's unlikely that something came
about "by chance" (with the precise likelihood and the concept of
"by chance" having more rigorous mathematical definitions than I
care to present in a blog comment). But if you try enough times,
even something that's "unlikely" can happen just by chance.
thoreau|9.17.07 @ 5:14PM|#
IIRC, global cooling was something that only a few scientists
ever did much on, and the media blew it out of proportion. However,
because of the attention it received, it will forever serve as
"proof" that those darn scientists just keep changing their minds
so they shouldn't be believed.
And yes, I'm aware that a popular news magazine ran a cover story
on global cooling in the 1970's. This is part of why you should
never trust popular news magazines over scientific
publications.
|9.17.07 @ 5:20PM|#
What is this "consensus" I keep hearing about? It keeps getting
filtered through the media and Al Gore's lips, so I don't know what
it really is. Is there really a consensus among climatologists that
the internal combustion engine is going to make the sea levels rise
eighteen feet over the next decade? Because that's what I keep
hearing.
I'm a skeptic of anthropogenic global warming, but I do believe
that the climate is changing, as it is in the nature of climates to
change. What scares me is not the CO2 emissions, but nutjob
politicians who get global warming religion and want to save us
from ourselves.
Fluffy|9.17.07 @ 5:20PM|#
I think this phenomenon is probably a function of the structure
of academia.
Persons chasing research grants and professorships have to publish.
This creates extreme pressure to find something new in research
areas that have already had quite a bit of the blood squeezed out
of their stones. One result of this is extreme hyperspecialization.
Another result of it is specious claims based on second-level
inference from old data.
I hate to give them any credit, but postmodernists have
demonstrated quite well how the reward system in academia in the
humanities warps the work produced in precisely these ways as well
[multiplying specializations and competition to find outlandish
theories that haven't already been overpublished and worn out].
It's a little surprising to see it on the sciences side, since
there should be more real grist for that side to work on, but it
isn't completely unexpected.
|9.17.07 @ 5:20PM|#
miscalculation, poor study design or self-serving data
analysis.
Poor study design is a particularly large problem in social
sciences. Unlike scientists and engineers who cut their teeth on
lots of small experiments before being allowed to move on to larger
ones, there are a number of paths to an advanced degree in social
sciences (and yes, some fields are better/worse than others) in
which the researcher has only a passing familiarity with
methodology, both from a theoretical and a practical
perspective.
Which is not to say that there aren't dolts and charlatans in
"real" science... just not as many.
Scientist|9.17.07 @ 5:23PM|#
Heresy! Apostasy!
Burn John Ioannidis for denying our peer-reviewed GOD!
Jesse Walker|9.17.07 @ 5:23PM|#
The hotter the field of research the more likely its
published findings should be viewed skeptically, he
determined.
How hot is Ioannidis' field of research?
|9.17.07 @ 5:23PM|#
Regular scientific claims, if they emcompass enough variables,
start being analyzed like social scientific claims. You necessarily
lose rigor. I know about regression analysis and all, but I still
shudder every time I see something that looks like r^2 of .65 and
someone saying "Wow! We really have something here!" As our friends
in criminology can attest, there are a lot of ways to regress a
cat.
I seem to recall that in my physics classes, publication of
propagated error analysis was the norm. You had to be able to
explain why your experimental results deviated from perfect
alignment with your proposed rule. Did people just stop doing
that?
Scientist|9.17.07 @ 5:23PM|#
Heresy! Apostasy!
Burn John Ioannidis for denying our peer-reviewed GOD!
Dr Mather|9.17.07 @ 5:28PM|#
"A new claim about a research finding is more likely to be
false than true."
The denier is a witch.
thoreau|9.17.07 @ 5:29PM|#
Burn John Ioannidis for denying our peer-reviewed
GOD!
Did you purchase carbon offsets for all the straw that you're
burning?
|9.17.07 @ 5:42PM|#
Has there been a study performed on "research paper corectness" vs year of publication? None was cited here. If said study hasn't been performed (I'm not volunteering) then IMHO Ioannidis' screed is a hypothesis at best.
|9.17.07 @ 5:42PM|#
How hot is Ioannidis' field of research?
Jesse, I don't think it's very hot, so we shouldn't be too
skeptical.
|9.17.07 @ 5:45PM|#
Look up Ioannidis' work on PubMED- he publishes in high-impact journals and is highly cited. His field- exposing publication bias- is hot because he made it hot. Plus he looks like Yanni.
|9.17.07 @ 6:16PM|#
Look up Ioannidis' work on PubMED- he publishes in
high-impact journals and is highly cited. His field- exposing
publication bias- is hot because he made it hot. Plus he looks like
Yanni.
So he's a skeptic. Good for him. I'm a skeptic too. Show me the
problem is increasing, till then, I'm skeptical.
Yes, I see the irony.
|9.17.07 @ 6:17PM|#
Like Thoreau said, testing multiple variables is a common way to inflate significance. Technically, a researcher is supposed to use a more rogrous standard of significance when he tests multiple variables, but few do in practice. Complex statistics also raise a red flag. No one bothers looking up the complex tests unless the simple tests already failed to find significant results.
|9.17.07 @ 6:24PM|#
IIRC, global cooling warming was
something that only a few scientists ever did much on, and the
media blew it out of proportion. However, because of the attention
it received, it will forever serve as "proof" that those darn
scientists just keep changing their minds so they shouldn't be
believed.
thoreau|9.17.07 @ 6:29PM|#
In 2050 the transhumanists will make it unnecessary for me to
dodge the oncoming glaciers. They'll just genetically modify me to
have the features of a polar bear, so it will all be good.
Just ask Ron Bailey.
:)
Douglas Gray|9.17.07 @ 7:38PM|#
Federal & corporate funding both corrupt. On a large scale, they corrupt absolutely.
|9.17.07 @ 8:18PM|#
Thoreau writes: "IIRC, global cooling was something that only a
few scientists ever did much on, and the media blew it out of
proportion. However, because of the attention it received, it will
forever serve as "proof" that those darn scientists just keep
changing their minds so they shouldn't be believed."
Plus, people's memories are probably conflating this with coverage
of 'nuclear winter', which also used to get a lot of publicity.
|9.17.07 @ 8:20PM|#
Correct me if I'm wrong, but isn't the point of peer reviewed articles that they are *not* the final word on anything, and may be wrong, and the results still need to be replicated by other researchers before it's really definitive?
|9.17.07 @ 8:22PM|#
Brandybuck wrote: " Is there really a consensus among
climatologists that the internal combustion engine is going to make
the sea levels rise eighteen feet over the next decade? Because
that's what I keep hearing."
Liar.
|9.17.07 @ 9:02PM|#
Statistically speaking, science suffers from an excess of
significance.
This jives with my own understanding of how academic science works.
Those who produce no results often fail to secure future funding.
Just as Those who design unmarketable products fial to get their
future endeavors funded.
thoreau|9.17.07 @ 9:05PM|#
Jon H-
Yep. Peer review is a low hurdle. It's a check to see whether the
paper gives a clear description of a well-designed and
well-executed study. The ultimate check is replication by
independent investigators. Since replication is only possible when
people know what they're supposed to be replicating, the goal of
peer review is to make sure that the methods, analysis, and results
are clearly described, and that any point not explained in the text
is explained in the cited references.
If somebody describes a sloppy experiment it's turned down, because
nobody has any interest in replicating sloppy work.
|9.17.07 @ 10:41PM|#
"There is an increasing concern that in modern research, false
findings may be the majority or even the vast majority of published
research claims," Dr. Ioannidis said.
The doc is pretty good at coughing up weasel words himself.
"Increasing concern." Where is it? Has anyone seen it? Can he offer
any quanitative evidence to support this statement? Or is this just
something he heard in the cafeteria? "False findings may be the
majority or even the vast majority." But, of course, maybe they
aren't the vast majority, or even the majority. Who knows? Not Dr.
Ioannidis.
|9.18.07 @ 7:00AM|#
One thing I would point out with regard to medical publications (although it may be true of other fields as well) is the bias in reporting an "effect" (i.e., something happened). Nobody wants to write up that the experiment they did gave no significant results. Therefore lots of articles that vitamin C does this, or brocoli does that, but not so many that no effect was found.
ed|9.18.07 @ 9:11AM|#
I'm no "scientist" but I'm pretty sure science is not about consensus. It's about correctly interpreting facts of reality. Individuals who cite "consensus" as proof of anything are merely partisans in a political game.
|9.18.07 @ 9:33AM|#
Describing the state of climatology as a consensus is another
way of saying that the thousands upon thousands of scientists,
acting independently and submitting their research for peer review,
keep coming to the same conclusion, test after test after test,
study after study after study.
The overwhelming consensus that exists is the RESULT of the
scientists drawing their conclusions from the data, not the CAUSE
of those conclusions.
There's also a consensus that the earth revolves around the sun.
This is because the evidence is overwhelming, not because people
are trying to fit in.
|9.18.07 @ 9:45AM|#
And of course as always with any tax payer funded research the
tag line that follows every report.
"More research will be required to get a better understanding,"
also read as Give us more money so we can spend the next 4 years
creating 1896 pages of useless data, oh and pay our mortgages.
thoreau|9.18.07 @ 9:57AM|#
The word "consensus" can mean different things in different
contexts. Some of those meanings have no relevance for science and
should be dismissed when brought up in the context of science (e.g.
"Everybody else thinks this way, so you should too."). Others are
very meaningful (e.g. "Many independent investigators, using a
variety of different methods, have achieved similar results in
their calculations, experiments, and observations.").
When you hear somebody use the word "consensus" in regards to
science, there are two things you can do:
1) You can assume that the word is being used in the worst way
possible, say "Science isn't about consensus!", and then ignore
everything that the person is saying. This is helpful if you don't
like what the person is saying. Unfortunately, however, science
isn't about whether you like something.
2) You can ask the person to clarify, and discuss whether the
person meant to make an argument from popularity or instead was
trying to point to a wide array of independent experimental
observations. The outcome of this discussion may be enlightening
for one or both participants.
Your choice.
GF|9.18.07 @ 10:33AM|#
I think it's time to end the Significance Game, and take prior
beliefs into account when faced with scientific results. A throw of
two sixes is "significant", but should you reject the hypothesis
that the dice are fair because of it?
Viva la Bayes Revolución already! (Start here:
yudkowsky.net/bayes/bayes.html )
ed|9.18.07 @ 11:03AM|#
Lip service to politically-correct scientific theories notwithstanding, it will be interesting to chart coastal real estate sales in the coming years. Will all the "believers" be moving inland? Will the "deniers" gobble up this windfall of vacated and ostensibly doomed property? Opining in a chatroom (sorry, a blog) is risk free. What people really think is revealed in how they spend their hard-earned cash.
|9.18.07 @ 11:21AM|#
ed,
No one denies the existence of hurricanes or earthquakes, but those
areas susceptible are not losing population. People are irrational
and act as if they will not be impacted.
ed|9.18.07 @ 12:19PM|#
People are irrational and act as if they will not be
impacted.
And it's up to the smart people to set them straight.
|9.18.07 @ 12:32PM|#
Well, we try, but people with enough motivation can convince
themselves of the most ridiculous things.
Such as, that we shouldn't believe scientists when they report on
their experiments.
Jed Rothwell|9.18.07 @ 3:09PM|#
Peer review does have shortcomings and it sometimes fails
drastically. For that matter, medical reviews sometimes fail to
catch disease, and bank examiners' reviews sometimes overlook
fraud. No system is infallible. On the whole, peer review is
beneficial. Cold fusion is an important case in point. In 1989,
many scientists jumped to the conclusion that cold fusion was a
mistake. Newspapers and magazines, which are not peer-reviewed,
attacked the discovery, and many continue to denigrate it today.
Some peer-reviewed journals such as Nature joined the bandwagon and
attacked the research.
However, editors at mainstream journals in chemistry,
electrochemistry and physics, such as Jap. J. Applied Physics and
Naturwissenschaften, reserved judgment and waited. After a few
years, serious replication experiments were completed and
peer-reviewed papers were published by researchers from national
laboratories and major universities worldwide. Eventually, hundreds
of positive papers were published, showing high-sigma excess heat,
tritium, neutrons, helium production and other nuclear effects.
This evidence leaves no doubt whatever that cold fusion is real.
Most journalists are unaware of this, so newspapers and other
secondary sources of information still report incorrect
information. If you want the facts you must read original source
scientific journals, and also official research papers published by
the Los Alamos, the U.S. Navy, the Italian National laboratories,
Bhabha Atomic Research Centre (BARC) and other trustworthy sources.
The internal standards at these laboratories are even more rigorous
than the peer-reviewed journals'.
- Jed Rothwell
Librarian, LENR-CANR.org
ed|9.18.07 @ 3:10PM|#
I will confess I'm happy all the smart people solved that ozone hole thing before it incinerated the planet. Funny how a problem that, according to consensus, would take a century to solve ""if we act right now" kinda fell off the radar screen after just a few years. Did we all get bored with the imminent disaster? Or was the predicted worldwide famine too distracting?
Jed Rothwell|9.18.07 @ 5:03PM|#
Ed wrote:
"I will confess I'm happy all the smart people solved that ozone
hole thing before it incinerated the planet. Funny how a problem
that, according to consensus, would take a century to solve 'if we
act right now' kinda fell off the radar screen after just a few
years."
The ozone hole is still a very serious problem. It would probably
be worse if we had not taken decisive action to eliminate some
ODS.
"Did we all get bored with the imminent disaster? Or was the
predicted worldwide famine too distracting?"
Worldwide famine is not "predicted" or "imminent." It is happening.
More people are starving today that at any time in history. Tens of
thousands die from malnutrition every week. It is true that the
percent of the population with adequate food is higher than it has
been in recorded history, but you should not make flippant jokes
about the two billion people who live in agony.
People who dismiss such problems remind me of the folks who were
upset after the year 2000, when serious Y2K computer disruptions
did not occur. I recall people said the Y2K problem was a hoax
invented by programmers, because nothing bad happened. Here are the
facts:
1. If corporations have not spent billions of dollars modifying the
programs in the last few years of the 20th century there would have
been a catastrophe.
2. Y2K was financial catastrophe. Fixing software cost far more
than it should have. The Y2K modifications should have been done
incrementally during routine maintenance starting in the 1980s.
Every program I wrote after 1985 was Y2K compliant. In many
corporations, the changes were made in a mad rush in the last few
years of the 20th century, which cost more than an orderly
change.
|9.18.07 @ 5:11PM|#
ed,
Are you under the impression that the Ozone Hole had closed?
It hasn't, although the bans on ozone depleting chemicals have made
a significant difference, and it is closing.
But thank you for brigning up such an convincing example of
environmental regulation solving a serious problem at minimal
cost.
|9.18.07 @ 5:26PM|#
The ozone hole was observed by satellites and other observation
methods that had not previously existed in the 1970s.
Since then it has gotten bigger some years and smaller
others.
It's entirely possible that its existence is completely natural and
is not related in any way to CFCs.
ed|9.18.07 @ 7:34PM|#
Global Warming is today's Cause Celebre. Get back to me in a couple years, when we'll be overwhelmingly distraught over the plight of Saturn's moons.
|9.19.07 @ 11:06PM|#
"The ozone hole was observed by satellites and other observation
methods that had not previously existed in the 1970s. "
On the other hand, there are effects that don't require satellites
to notice - increased sunburn and skin cancer in areas under the
hole.
|9.20.07 @ 11:33AM|#
The hole is over Ant-fucking-artica. Who the fuck is getting
sunburned?
If your taking about Australia, it might just have something to do
with increased population.
They always had high skin cancer rates.
Site comments/questions:
Media Inquiries and Reprint Permissions:
(310) 367-6109
Editorial & Production Offices:
3415 S. Sepulveda Blvd.
Suite 400
Los Angeles, CA 90034
(310) 391-2245