Science

Are Most Scientific Results Bunk?

|

I recently enjoyed listening to Stanford University statistician John Ioannidis and University of Virginia psychologist Brian Nosek talk about how to make scientific evidence more reliable. Ioannides gained some well-deserved fame with his 2005 article in PLoS One, "Why Most Published Research Findings Are False," and Brian Nosek has just established the Center for Open Science which is offering its Open Science Framework that aims to improve the validity of scientific research.

In a recent article, "Evaluation of Very Large Treatment Effects of Medical Interventions," in the Journal of the American Medical Association, Ioannid1s and his colleagues combed through 85,000 medical interventions collected in the Cochrane Database of Systematic Reviews seeking to uncover highly effective treatments. What they found is that treatments that supposedly produce very large benefits (odds ratio greater than 5) were almost always found initially in small studies and that when they were replicated in larger studies the benefits became relatively modest. In the end only one treatment was found to provide a major benefit, e.g., supplying extracorporeal oxgyen to premature babies with severe respiratory failure. Last year, Nature reported the shocking finding that nine out of 10 preclinical peer-reviewed cancer research studies cannot be replicated.

Another big problem is the bias toward publishing positive results, while sticking negative results in the file drawer. In an interesting 2010 study published in PLoS One, University of Edinburgh researcher Daniele Fanelli found that as the science under consideration got "softer' the more positive results were reported. From the abstract:

This study analysed 2434 papers published in all disciplines and that declared to have tested a hypothesis. It was determined how many papers reported a "positive" (full or partial) or "negative" support for the tested hypothesis. If the hierarchy hypothesis is correct, then researchers in "softer" sciences should have fewer constraints to their conscious and unconscious biases, and therefore report more positive outcomes. Results confirmed the predictions at all levels considered: discipline, domain and methodology broadly defined. Controlling for observed differences between pure and applied disciplines, and between papers testing one or several hypotheses, the odds of reporting a positive result were around 5 times higher among papers in the disciplines of Psychology and Psychiatry and Economics and Business compared to Space Science, 2.3 times higher in the domain of social sciences compared to the physical sciences, and 3.4 times higher in studies applying behavioural and social methodologies on people compared to physical and chemical studies on non-biological material. In all comparisons, biological studies had intermediate values.

Last week, the Economist had a terrific article outlining the problems with lack of replicability and lax peer review in science, not least of which is that those problems mislead subsequent research efforts and is huge waste of money and talent. From the Economist:

The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America's National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.

The whole Economist article is well worth your time. Nosek's Open Science Framework project seems like a promising way to nudge researchers toward greater transparency and less data dredging. Through the system researchers can obtain "badges" for project pre-registration, open data, and open materials. Presumably these badges will help persuade journal editors to be more likely to publish such studies and thus encourage better research practices. 

NEXT: Barack Obama Giving Speech on Obamacare in Boston Same Day as World Series Game

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. the public part of the process seems to have failed

    Of course. As long as any science is being politicized, it cannot possibly be trusted. And so much that we see is politicized, “climate change” being one of the best examples of this. The involvement of government and politicization distort the scientific process too much for it to be even remotely reliable.

    The best way to use research is as a negative tool. For instance, people claimed that call phones could give you cancer. If anyone could have produced a study proving that, they would have in an instant. The fact that no one did tells you a lot about the claims of causing cancer. Or claims that organic food is better for you. If the organic proponents could get even one study that conclusively proved that organic is better, they would be screaming it from the rooftops. The fact that they aren’t tells you a lot about their claims.

    1. Yep. Keep this article handy to throw in the face of anyone stupid enough to use the line ‘the science is settled!’.

      1. This article is not a license to believe whatever the fuck you want to believe. Scientists didn’t politicize this subject, oil and coal interests did.

        1. ^^ Speaking of anyone stupid enough…

        2. You’ve very effectively expressed the third grade girl position in the argument. Bravo.

        3. Oil and coal interests politicize science on abortion, creation science, supply side economics, etc.? Suffering from tunnel vision are we?

        4. Well, yes, it is true that oil and coal companies have politicized the science involved in their field, but so have all the opponents of oil and coal. This is true of every field which relates to a political issue (in other words, practically every field of science). And even scientists who are not working for a particular interest group may still have political or moral opinions which bias their research.

          1. Science is the best method devised by humans for removing bias.

            Profit motive on the other hand, is inherently ideologically motivational. A false equivalence is tantamount to a lie.

            1. Tony, Tony, Tony.

              As an economy becomes less influenced and distorted by politics (government) profit is determined by producing more efficient products and services than your competitors.

              Less protectionism = more competition.

              More competition = more proportional relationship between effectiveness of product and profit.

              Pretty easy to understand, if you actually care about prosperity and quality of life for as many people as possible.

              1. As an economy becomes less influenced and distorted by politics (government) profit is determined by producing more efficient products and services than your competitors.

                I know you need this to be true, but it just isn’t necessarily. Absent government intervention, profit can become the product of monopoly and collusion. It can be the product of being more willing to sell dangerous products than your competitor. Platitudes do not a political system make, and there are no such things as unicorns.

                1. And would selling dangerous products benefit a company that has competitors?

                  1. If people only care about profit would competitors immediately start making a safer product since it would make more money?

                    You cant have it both ways. If all people are greedy, than profit is determined by please customers (who only care about getting the best deal) and therefore, government officials too are greedy and therefore cannot be trusted to act in your self interest.

            2. Also, best method to remove bias does not equal 100% effective method of reducing bias. We are human, afterall, and are incredibly capable of overcoming good ideas and methods with our immense desire to “win” or be proven “right”.

        5. Scientists didn’t politicize this subject

          The second someone suggested that everyone should somehow be forced to reduce CO2 emissions, that’s when the issue became politicized.

          1. The second someone suggested that everyone should somehow be forced to reduce CO2 emissions, that’s when the issue became politicized.

            Bravo, sir!
            Given that CO2 has been shown to be aerial plant food, and that it has greened the Earth (especially in places like the African deserts) over the last 30 years, anyone who wants to reduce CO2 emissions clearly hates plants.

            1. Which other form of food is considered beneficial no matter how much is consumed?

          2. Well sure, but that’s no excuse for spreading lies. There should be a political debate. It should be about how to solve the problem, not whether the problem is real. Only one set of interests benefits from that kind of debate–the ones in favor of doing nothing.

            Even if you happen to agree with that policy approach, you don’t get to pretend facts aren’t facts in order to push it.

        6. Hey, Tony, where’ve you been? We’ve missed your input on all those threads showing your prized piece of legislation to be imploding. I’m glad you’ve chosen to come back and comment on something finally, although admittedly without an argument of any kind.

          1. Sometimes I need a vacation from crazy.

        7. Hey, Tony… can you actually walk when your knees jerk so often?

  2. She was very delighted with this new idea, and went to her professor. And his reply was, no, you cannot do that, because the experiment has already been done and you would be wasting time. This was in about 1947 or so, and it seems to have been the general policy then to not try to repeat psychological experiments, but only to change the conditions and see what happened.

    Nowadays, there’s a certain danger of the same thing happening, even in the famous field of physics. I was shocked to hear of an experiment being done at the big accelerator at the National Accelerator Laboratory, where a person used deuterium. In order to compare his heavy hydrogen results to what might happen with light hydrogen, he had to use data from someone else’s experiment on light hydrogen, which was done on different apparatus. When asked why, he said it was because he couldn’t get time on the program (because there’s so little time and it’s such expensive apparatus) to do the experiment with light hydrogen on this apparatus because there wouldn’t be any new result. And so the men in charge of programs at NAL are so anxious for new results, in order to get more money to keep the thing going for public relations purposes, they are destroying–possibly–the value of the experiments themselves, which is the whole purpose of the thing. It is often hard for the experimenters there to complete their work as their scientific integrity demands. …

    1. I looked up the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or being very careful. They just went right on running the rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn’t discover anything about the rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic example of cargo cult science.

      http://neurotheory.columbia.ed….._cult.html

      1. Good ol’ Richard Feynman. Still decades ahead of his time.

    2. “If it disagrees with experiment, it’s wrong. And that simple statement is the key to science.”

      1. I remember Tony losing his shit the first time I linked to that video.

        He shrieked that we weren’t allowed to use Feynmann.

        It was delicious. 😀

        1. Um…not allowed to use Feynman? What?

          1. Nothing pisses me off more than people invested in a particular dogma, which is overwhelmingly contradicted by evidence, whoring out science and scientists to back up their bullshit. It’s not just Feynman, add Galileo to the list.

            1. I didn’t give you permission to speak to me, you fucking clown. Fuck off back where you came from.

            2. The irony – it burns

              1. Kindly point me to a reliable source that explains how there is no evidence for human-caused climate change.

                1. Actually, Tonykins, we are refering to the falsificatoion of CAGW, ie that humans are triggering catastrophical warming.

                  And you can look no further than the IPCC for your ‘scientifically’ reputable source. They admit that CAGW ain’t happening.

                  See table 12.4 from the latest IPCC assessment.

                  1. I gather then that you have always accepted the A and the GW, and that this isn’t another mind-numbingly obvious moving of goalposts similar to when creationists had to resort to babbling about bacterial flagella before they were finally tossed away into the dustbin of irrelevance (except in Texas).

                    I suppose it depends on how you define catastrophic, but I don’t see why rising sea levels, ocean acidification, increased extreme weather events, and a general rapid disruption of what has been the climate status quo for the entirety of the existence of human civilization (all already evidenced) are no big deal.

                    1. Details play hell with an otherwise pretty theory:

                      rising sea levels

                      …have been in process well in excess of 10,000 years. Hard to blame that on people.

                      increased extreme weather events

                      …which is why the #2 and #3 spots for the most Atlantic basin tropical cyclones are 1933 and 1887? And the record earliest formation is January 3, 1938?

                      So, how much AGW are you asserting had happened in the 1880s and 1930s?

                    2. So the greenhouse effect isn’t real? Or what’s your point?

                      I’m just taking from the IPCC report.

                2. I was refering to you generally suggesting that you don’t stick to your dogma in the face of emperical evidence to the contrary. You are at least as guilty of that compared to most people.

          2. I can’t find it. Unfortunate because it was really funny.

            I think it was spring of this year.

  3. Hmm, let’s see:

    1) Private research conducted by for-profit organizations hoping to get approval to sell products may overstate positive results and downplay negative results.

    2) Public research conducted by not-for-profit organizations hoping to get future funding for additional research may overstate positive results and downplay negative results.

    Well, we can at least kill off one of the problems by shutting off the public spigot.

  4. Are Most Scientific Results Bunk?

    I don’t know, we should conduct a scientific study.

    1. “I don’t know, we should conduct a scientific study.”

      And what would you like the answer to be, Mr. President?

  5. Maybe if we didn’t have the FDA and billions of dollars were not riding on proving that something is safe, there would be more integrity?

  6. I propose an Iron Law that “science” be used exclusively with ellipses going forward to put it in the proper context.

    Ex 1: Climate “science” says the earth is both burning up and freezing at the same time.
    -or-
    Ex 2: The “scientific” community has determined that hydrofracking causes cancer in children and creates devastating sinkholes in places like Peru.

  7. So, the science is still settled, right?

    1. The science will be settled as long as the funding continues…

  8. I don’t believe it
    There she goes again
    She’s tidied up and I can’t find anything
    All my tubes and wires
    And careful notes
    And antiquated notions

    But – it’s poetry in motion
    And when she turned her tender eyes to me
    As deep as any ocean
    As sweet as any harmony
    Mmm – but she blinded me with science
    She blinded me with science
    And failed me in geometry

  9. Most, I would say a large majority, of research, is funded by someone or some organization, private interest or government. And that someone or something is not typically the person or organization doing the research.

    Most researchers need grant money to pay for their research. How much percentage of that research do you assume starts out with the organization paying for the research making a statement to the person or organization doing the research, something to the effect of ‘and this is the results we are looking for.’? I am going to say most of the time, that happens. And what happens to that researcher when they don’t find the results that their sponsor is looking for?

  10. Most scientific results are perfectly fine. The problem is in the cargo cults.

  11. I am most disgusted by the pandemic of rationalism in modern science. Rationalism has always been the bane of the humanities, but to see physics taken over by rationalists (thanks Einstein) should be evidence that something is very, very wrong in the philosophy of science; oh wait, scientists don’t use evidence or philosophy anymore, so nevermind.

    This “hypothetico-deductive” method is bullshit; no, you don’t have to go through the painstaking process of observation and inductive proof, you just rationalize a theory and then throw it at the wall of reality to see if it sticks. Isaac Newton was an old-fashioned fool who believed in objective reality; enlightened spirits know that reality is a synthetic product of consciousness.

    1. Oh no not the bizzaro Objectivist Jihad against quantum physics!

      1. Is that what Liberarius is talking about. Because I had no cluse

        1. or clue either

        2. Because I had no cluse

          You damned communist!

      2. It’s quantum physics that are *transparently* rationalistic and “bizarro”; observe the merger between quantum theory and Zen Buddhism (lolzolzolz), two peas in the same mystic pod. It is no accident that the Catholic Church was so enthusiastic to embrace quantum theory.

        The mathematical formulas work–that much is obvious. But they absolutely do not prove the existence of multiple dimensions, or the “Big Bang”, or god, or that entities can randomly pop into and out of existence, or that the laws of identity and causality have been refuted. To the extent that the formulas work, they are consonant with these laws.

  12. But the science is settled that if she weighs as much as a duck she is a witch, right?

  13. The problem with most research studies is that they rely on null hypothesis significance testing. Nate Silver has a good description in his book but in a nutshell, there is no plausibility placed on the hypothesis being tested in a frequentist framework. Many statisticians are advocating that Bayesian methods be used to overcome this liability.

    1. Bayseanism might be useful, but merely increasing the required standard of significance would itself be able to get rid of a lot of the shit. Demand three sigma instead of two in a frequentist framework, and a lot of the nonsense falls right out.

  14. Another big problem is the bias toward publishing positive results, while sticking negative results in the file drawer.

    I actually once read, in a sort of ‘how to be a scientist’ oldschool publication, that one shouldn’t publish negative findings because it would discourage work on that particular area…which is mystifying.

    1. I would hope it discourages research groups all over the world from doing the same experiment over and over again never realizing that it will get them nowhere.

  15. But patents. We’d get a lot better science with the open science framework. But Universities love their IP, and spin offs wave their patents at investors to get funds.

    Also small research groups need to be fairly protective and secretive or the big groups with more resources will eat them for lunch.

  16. I accepted a PhD candidate positions today (in nuclear engineering). I was really trying to avoid continuing on to a PhD but the project offered to me will directly affect the nuclear industry and is much more of an engineering project than a theoretical thesis.

    I already feel the pressure to produce positive results for my project. I would also much rather come up with positive results than negative. However, my project has a good amount of design flexibility inherent to it so finding results that offer benefits to both safety and economics should hopefully be possible.

    1. Yeah funny how mine basically became an engineering project too, computational in my case.

      Our funder would like to see positive results but if they are negative would just use them to design a new product. So I don’t feel much pressure…today.

      1. One of the things I found out when I was playing the NSF/NIH grant game was that they officially were looking to fund risky projects- I was stupid enough to believe that and put in proposals for really novel research directions. Bounced back faster than a Superball.

        Then I learned how to play the game and pulled in $4MM in two years. Show something that not only has no risk, but is accompanied by “preliminary data” which show that you already have the outcome nailed, you just need to repeat it a few times. Four “Confirm that…” to one “Determine that…” in the Specific Aims or Research Goals section and grant riches are yours. The grant officer is judged on successful research programs, not “interesting idea but it didn’t pan out” ones.

        1. The trick is to write proposals for research that you’ve already done, at least mostly done.

          1. It’s similar to the “one-ahead” in mentalism. You spend the grant time getting “preliminary data” to snag the next grant.

  17. NPR did a story on a science journalist who tested online journals by submitting a fake journal article to see they would be published.

    http://www.npr.org/blogs/healt…..-for-a-fee

    “In the end, the paper’s fictitious authors got 157 acceptance letters and 98 rejections ? a score of 61 percent. ‘That’s way higher than I expected,’ Bohannan says. ‘I was expecting 10 or 15 percent, or worst case, a quarter accepted.'”

    I don’t think it was Ioannidis intent to throw all of science into question and make science a matter of opinion as some are treating it. Instead he was troubled by the lack of control for bias and reproducible results. The problem isn’t the process of science, but rather the lack of following the process of science.

Please to post comments

Comments are closed.