Science: Mostly A Pack Of Lies
….says scientist John Ioannidis. The Wall Street Journal summarizes his findings:
In a series of influential analytical reports, [Ioannidis] has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.
These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
…….
Statistically speaking, science suffers from an excess of significance. Overeager researchers often tinker too much with the statistical variables of their analysis to coax any meaningful insight from their data sets. "People are messing around with the data to find anything that seems significant, to show they have found something that is new and unusual," Dr. Ioannidis said.
Editor's Note: As of February 29, 2024, commenting privileges on reason.com posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary period. Subscribe here to preserve your ability to comment. Your Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the digital edition and archives of Reason magazine. We request that comments be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Please
to post comments
Hey...how bout that global warming....?!?
Global warming is probably one of those areas that has (or at least has had) so many scientists scrutinizing the problem, and one another's work, so closely that the consensus is not far off the mark.
Global warming is cooling was probably one of those areas that has (or at least has had) had so many scientists scrutinizing the problem, and one another's work, so closely that the consensus is seemed at the time not far off the mark to the majority of the government-funded scientists jumping on the bandwagon that promised additional funding for climate research.
Fixed it.
If this is the guy that I'm thinking of, he has some sobering observations on the fact that if you try enough variables eventually you'll find one that, by chance, is statistically significant.
You may be thinking "What? I thought statistical significance is all about something not being about chance!" Actually, statistical significance means that it's unlikely that something came about "by chance" (with the precise likelihood and the concept of "by chance" having more rigorous mathematical definitions than I care to present in a blog comment). But if you try enough times, even something that's "unlikely" can happen just by chance.
IIRC, global cooling was something that only a few scientists ever did much on, and the media blew it out of proportion. However, because of the attention it received, it will forever serve as "proof" that those darn scientists just keep changing their minds so they shouldn't be believed.
And yes, I'm aware that a popular news magazine ran a cover story on global cooling in the 1970's. This is part of why you should never trust popular news magazines over scientific publications.
"How the hell did you get in here?!?"
"Science...Fiction!"
What is this "consensus" I keep hearing about? It keeps getting filtered through the media and Al Gore's lips, so I don't know what it really is. Is there really a consensus among climatologists that the internal combustion engine is going to make the sea levels rise eighteen feet over the next decade? Because that's what I keep hearing.
I'm a skeptic of anthropogenic global warming, but I do believe that the climate is changing, as it is in the nature of climates to change. What scares me is not the CO2 emissions, but nutjob politicians who get global warming religion and want to save us from ourselves.
I think this phenomenon is probably a function of the structure of academia.
Persons chasing research grants and professorships have to publish. This creates extreme pressure to find something new in research areas that have already had quite a bit of the blood squeezed out of their stones. One result of this is extreme hyperspecialization. Another result of it is specious claims based on second-level inference from old data.
I hate to give them any credit, but postmodernists have demonstrated quite well how the reward system in academia in the humanities warps the work produced in precisely these ways as well [multiplying specializations and competition to find outlandish theories that haven't already been overpublished and worn out]. It's a little surprising to see it on the sciences side, since there should be more real grist for that side to work on, but it isn't completely unexpected.
miscalculation, poor study design or self-serving data analysis.
Poor study design is a particularly large problem in social sciences. Unlike scientists and engineers who cut their teeth on lots of small experiments before being allowed to move on to larger ones, there are a number of paths to an advanced degree in social sciences (and yes, some fields are better/worse than others) in which the researcher has only a passing familiarity with methodology, both from a theoretical and a practical perspective.
Which is not to say that there aren't dolts and charlatans in "real" science... just not as many.
Heresy! Apostasy!
Burn John Ioannidis for denying our peer-reviewed GOD!
The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
How hot is Ioannidis' field of research?
Yeah, What Fluffy said.
Regular scientific claims, if they emcompass enough variables, start being analyzed like social scientific claims. You necessarily lose rigor. I know about regression analysis and all, but I still shudder every time I see something that looks like r^2 of .65 and someone saying "Wow! We really have something here!" As our friends in criminology can attest, there are a lot of ways to regress a cat.
I seem to recall that in my physics classes, publication of propagated error analysis was the norm. You had to be able to explain why your experimental results deviated from perfect alignment with your proposed rule. Did people just stop doing that?
Heresy! Apostasy!
Burn John Ioannidis for denying our peer-reviewed GOD!
"A new claim about a research finding is more likely to be false than true."
The denier is a witch.
Burn John Ioannidis for denying our peer-reviewed GOD!
Did you purchase carbon offsets for all the straw that you're burning?
Has there been a study performed on "research paper corectness" vs year of publication? None was cited here. If said study hasn't been performed (I'm not volunteering) then IMHO Ioannidis' screed is a hypothesis at best.
That's why informed citizens get their answers from Bibleman.
How hot is Ioannidis' field of research?
Jesse, I don't think it's very hot, so we shouldn't be too skeptical.
Look up Ioannidis' work on PubMED- he publishes in high-impact journals and is highly cited. His field- exposing publication bias- is hot because he made it hot. Plus he looks like Yanni.
Look up Ioannidis' work on PubMED- he publishes in high-impact journals and is highly cited. His field- exposing publication bias- is hot because he made it hot. Plus he looks like Yanni.
So he's a skeptic. Good for him. I'm a skeptic too. Show me the problem is increasing, till then, I'm skeptical.
Yes, I see the irony.
Like Thoreau said, testing multiple variables is a common way to inflate significance. Technically, a researcher is supposed to use a more rogrous standard of significance when he tests multiple variables, but few do in practice. Complex statistics also raise a red flag. No one bothers looking up the complex tests unless the simple tests already failed to find significant results.
IIRC, global cooling warming was something that only a few scientists ever did much on, and the media blew it out of proportion. However, because of the attention it received, it will forever serve as "proof" that those darn scientists just keep changing their minds so they shouldn't be believed.
In 2050 the transhumanists will make it unnecessary for me to dodge the oncoming glaciers. They'll just genetically modify me to have the features of a polar bear, so it will all be good.
Just ask Ron Bailey.
🙂
Federal & corporate funding both corrupt. On a large scale, they corrupt absolutely.
Thoreau writes: "IIRC, global cooling was something that only a few scientists ever did much on, and the media blew it out of proportion. However, because of the attention it received, it will forever serve as "proof" that those darn scientists just keep changing their minds so they shouldn't be believed."
Plus, people's memories are probably conflating this with coverage of 'nuclear winter', which also used to get a lot of publicity.
Correct me if I'm wrong, but isn't the point of peer reviewed articles that they are *not* the final word on anything, and may be wrong, and the results still need to be replicated by other researchers before it's really definitive?
Brandybuck wrote: " Is there really a consensus among climatologists that the internal combustion engine is going to make the sea levels rise eighteen feet over the next decade? Because that's what I keep hearing."
Liar.
Statistically speaking, science suffers from an excess of significance.
This jives with my own understanding of how academic science works. Those who produce no results often fail to secure future funding. Just as Those who design unmarketable products fial to get their future endeavors funded.
Jon H-
Yep. Peer review is a low hurdle. It's a check to see whether the paper gives a clear description of a well-designed and well-executed study. The ultimate check is replication by independent investigators. Since replication is only possible when people know what they're supposed to be replicating, the goal of peer review is to make sure that the methods, analysis, and results are clearly described, and that any point not explained in the text is explained in the cited references.
If somebody describes a sloppy experiment it's turned down, because nobody has any interest in replicating sloppy work.
Hooray! Prayer it is then.
"There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said.
The doc is pretty good at coughing up weasel words himself. "Increasing concern." Where is it? Has anyone seen it? Can he offer any quanitative evidence to support this statement? Or is this just something he heard in the cafeteria? "False findings may be the majority or even the vast majority." But, of course, maybe they aren't the vast majority, or even the majority. Who knows? Not Dr. Ioannidis.
One thing I would point out with regard to medical publications (although it may be true of other fields as well) is the bias in reporting an "effect" (i.e., something happened). Nobody wants to write up that the experiment they did gave no significant results. Therefore lots of articles that vitamin C does this, or brocoli does that, but not so many that no effect was found.
I'm no "scientist" but I'm pretty sure science is not about consensus. It's about correctly interpreting facts of reality. Individuals who cite "consensus" as proof of anything are merely partisans in a political game.
Describing the state of climatology as a consensus is another way of saying that the thousands upon thousands of scientists, acting independently and submitting their research for peer review, keep coming to the same conclusion, test after test after test, study after study after study.
The overwhelming consensus that exists is the RESULT of the scientists drawing their conclusions from the data, not the CAUSE of those conclusions.
There's also a consensus that the earth revolves around the sun. This is because the evidence is overwhelming, not because people are trying to fit in.
And of course as always with any tax payer funded research the tag line that follows every report.
"More research will be required to get a better understanding," also read as Give us more money so we can spend the next 4 years creating 1896 pages of useless data, oh and pay our mortgages.
The word "consensus" can mean different things in different contexts. Some of those meanings have no relevance for science and should be dismissed when brought up in the context of science (e.g. "Everybody else thinks this way, so you should too."). Others are very meaningful (e.g. "Many independent investigators, using a variety of different methods, have achieved similar results in their calculations, experiments, and observations.").
When you hear somebody use the word "consensus" in regards to science, there are two things you can do:
1) You can assume that the word is being used in the worst way possible, say "Science isn't about consensus!", and then ignore everything that the person is saying. This is helpful if you don't like what the person is saying. Unfortunately, however, science isn't about whether you like something.
2) You can ask the person to clarify, and discuss whether the person meant to make an argument from popularity or instead was trying to point to a wide array of independent experimental observations. The outcome of this discussion may be enlightening for one or both participants.
Your choice.
I think it's time to end the Significance Game, and take prior beliefs into account when faced with scientific results. A throw of two sixes is "significant", but should you reject the hypothesis that the dice are fair because of it?
Viva la Bayes Revoluci?n already! (Start here: yudkowsky.net/bayes/bayes.html )
Lip service to politically-correct scientific theories notwithstanding, it will be interesting to chart coastal real estate sales in the coming years. Will all the "believers" be moving inland? Will the "deniers" gobble up this windfall of vacated and ostensibly doomed property? Opining in a chatroom (sorry, a blog) is risk free. What people really think is revealed in how they spend their hard-earned cash.
ed,
No one denies the existence of hurricanes or earthquakes, but those areas susceptible are not losing population. People are irrational and act as if they will not be impacted.
People are irrational and act as if they will not be impacted.
And it's up to the smart people to set them straight.
Well, we try, but people with enough motivation can convince themselves of the most ridiculous things.
Such as, that we shouldn't believe scientists when they report on their experiments.
Peer review does have shortcomings and it sometimes fails drastically. For that matter, medical reviews sometimes fail to catch disease, and bank examiners' reviews sometimes overlook fraud. No system is infallible. On the whole, peer review is beneficial. Cold fusion is an important case in point. In 1989, many scientists jumped to the conclusion that cold fusion was a mistake. Newspapers and magazines, which are not peer-reviewed, attacked the discovery, and many continue to denigrate it today. Some peer-reviewed journals such as Nature joined the bandwagon and attacked the research.
However, editors at mainstream journals in chemistry, electrochemistry and physics, such as Jap. J. Applied Physics and Naturwissenschaften, reserved judgment and waited. After a few years, serious replication experiments were completed and peer-reviewed papers were published by researchers from national laboratories and major universities worldwide. Eventually, hundreds of positive papers were published, showing high-sigma excess heat, tritium, neutrons, helium production and other nuclear effects. This evidence leaves no doubt whatever that cold fusion is real. Most journalists are unaware of this, so newspapers and other secondary sources of information still report incorrect information. If you want the facts you must read original source scientific journals, and also official research papers published by the Los Alamos, the U.S. Navy, the Italian National laboratories, Bhabha Atomic Research Centre (BARC) and other trustworthy sources. The internal standards at these laboratories are even more rigorous than the peer-reviewed journals'.
- Jed Rothwell
Librarian, LENR-CANR.org
I will confess I'm happy all the smart people solved that ozone hole thing before it incinerated the planet. Funny how a problem that, according to consensus, would take a century to solve ""if we act right now" kinda fell off the radar screen after just a few years. Did we all get bored with the imminent disaster? Or was the predicted worldwide famine too distracting?
Ed wrote:
"I will confess I'm happy all the smart people solved that ozone hole thing before it incinerated the planet. Funny how a problem that, according to consensus, would take a century to solve 'if we act right now' kinda fell off the radar screen after just a few years."
The ozone hole is still a very serious problem. It would probably be worse if we had not taken decisive action to eliminate some ODS.
"Did we all get bored with the imminent disaster? Or was the predicted worldwide famine too distracting?"
Worldwide famine is not "predicted" or "imminent." It is happening. More people are starving today that at any time in history. Tens of thousands die from malnutrition every week. It is true that the percent of the population with adequate food is higher than it has been in recorded history, but you should not make flippant jokes about the two billion people who live in agony.
People who dismiss such problems remind me of the folks who were upset after the year 2000, when serious Y2K computer disruptions did not occur. I recall people said the Y2K problem was a hoax invented by programmers, because nothing bad happened. Here are the facts:
1. If corporations have not spent billions of dollars modifying the programs in the last few years of the 20th century there would have been a catastrophe.
2. Y2K was financial catastrophe. Fixing software cost far more than it should have. The Y2K modifications should have been done incrementally during routine maintenance starting in the 1980s. Every program I wrote after 1985 was Y2K compliant. In many corporations, the changes were made in a mad rush in the last few years of the 20th century, which cost more than an orderly change.
ed,
Are you under the impression that the Ozone Hole had closed?
It hasn't, although the bans on ozone depleting chemicals have made a significant difference, and it is closing.
But thank you for brigning up such an convincing example of environmental regulation solving a serious problem at minimal cost.
The ozone hole was observed by satellites and other observation methods that had not previously existed in the 1970s.
Since then it has gotten bigger some years and smaller others.
It's entirely possible that its existence is completely natural and is not related in any way to CFCs.
Global Warming is today's Cause Celebre. Get back to me in a couple years, when we'll be overwhelmingly distraught over the plight of Saturn's moons.
"The ozone hole was observed by satellites and other observation methods that had not previously existed in the 1970s. "
On the other hand, there are effects that don't require satellites to notice - increased sunburn and skin cancer in areas under the hole.
The hole is over Ant-fucking-artica. Who the fuck is getting sunburned?
If your taking about Australia, it might just have something to do with increased population.
They always had high skin cancer rates.