Reason.com

Free Minds & Free Markets

Broken Science

What happens when cancer doctors, psychologists, and drug developers can't rely on each other's research?

Jason KeislingJason KeislingNullius in verba is the motto of one of the earliest scientific associations-the Royal Society, founded in 1663. Broadly translated, the phrase means "Don't take anybody's word for it."

You know how it's supposed to work: A scientist should ideally be able to do the same experiment as any other scientist and get similar results. As researchers check and recheck each other's findings, the sphere of knowledge expands. Replication is the path to scientific advancement.

Some 15 million researchers published more than 25 million scientific papers between 1996 and 2011. Among them were several casting doubt on the veracity and reliability of the rest—suggesting that even studies published in gold-standard journals by researchers from top-tier institutions are far more likely than anyone previously realized to be false, fudged, or flukey. The upshot is that many researchers have come to believe that science is badly battered, if not broken.

Everything We Know Is Wrong?

The Stanford statistician John Ioannidis sounded the alarm about our science crisis 10 years ago. "Most published research findings are false," Ioannidis boldly declared in a seminal 2005 PLOS Medicine article. What's worse, he found that in most fields of research, including biomedicine, genetics, and epidemiology, the research community has been terrible at weeding out the shoddy work largely due to perfunctory peer review and a paucity of attempts at experimental replication. Ioannidis showed, for instance, that about one-third of the results of highly cited original clinical research studies were shown to be wrong or exaggerated by subsequent research. "For many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias," he argued. Today, he says science is still wracked by the reproducibility problem: "In several fields, it is likely that most published research is still false."

Initially, some researchers argued that Ioannidis' claims were significantly overstated. "We don't think the system is broken and needs to be overhauled," declared New England Journal of Medicine editor Jeffrey Drazen in The Boston Globe in 2005. But his analyses have sparked a vast and ongoing reassessment of how science is done. Once other scientists started looking into the question, they found the same alarming trend everywhere.

In 2012, researchers at the pharmaceutical company Amgen reported in Nature that they were able to replicate the findings of only six out of 53 (11 percent) landmark published preclinical cancer studies. Preclinical studies test a drug, a procedure, or another medical treatment in animals as precursors to human trials. In 2011, researchers at Bayer Healthcare reported that they could not replicate 43 of the 67 published preclinical studies that the company had been relying on to develop cancer and cardiovascular treatments and diagnostics. Ioannidis estimates that "in biomedical sciences, non-replication rates that have been described range from more than 90 percent for observational associations (e.g., nutrient X causes cancer Y), to 75–90 percent for preclinical research (trying to find new drug targets)."

The mounting evidence that most scientific findings are false provoked a rash of worried headlines, including "How Science Is Broken" at Vox in 2015; "Why medical clinical trials are so wrong so often" in The Washington Post in 2015; "The Truth Is Many Scientific Studies Can't Be Trusted" at Business Insider in 2012; "Lies, Damned Lies, and Medical Science" in The Atlantic in 2010; and "Most Science Studies Appear to Be Tainted by Sloppy Analysis" in The Wall Street Journal in 2007.

In April 2015, the editor of the prestigious medical journal The Lancet reported that the participants in a recent conference believed "much of the scientific literature, perhaps half, may simply be untrue." In August, the journal Science reported that only one-third of 100 psychological studies published in three leading psychology journals could be adequately replicated. In October, a panel of senior researchers convened by the British Academy of Medical Sciences (BAMS) issued a major report on research reproducibility indicating that the false discovery rate in some areas of biomedicine could be as high as 69 percent.

If the rate of false positives is as high as feared, the Oxford researchers Ian Chalmers and Paul Glasziou suggest that as much as 85 percent of the resources devoted to biomedical research are being wasted. Globally, that amounts to about $200 billion every year.

In a June 2015 article for PLOS Biology, Leonard Freedman of the Global Biological Standards Institute and his colleagues note that published estimates for the reproducibility of preclinical research range from 51 percent to 89 percent. They estimate that at least half of all U.S. preclinical biomedical research funding—about $28 billion annually—is therefore squandered. As the colossal failure to replicate prominent cancer studies indicates, this wasted research results in treatments not developed and cures not found.

Venture capital firms now take it for granted, according to SciBX: Science-Business eXchange, that 50 percent of published academic studies cannot be replicated by industrial laboratories. Before investing in biomedical startups, they often hedge against "academic risk" by hiring contract research organizations to vet the science. This slows down the process of translating genuine discoveries into new products.

The invention of the scientific process during the past two centuries is arguably humanity's greatest intellectual achievement. Science and the technological progress it fosters have dramatically lengthened life spans, lessened the burdens of disease, reduced ignorance, and eased the hardships of work and daily life. We are living in a time of technological marvels, with advances like CRISPR gene-editing being used to bring back extinct mammoths; lithium-air batteries that store 10 times more energy than conventional lithium-ion batteries; mitochondrial transfers that create healthy babies who have three genetic parents; Ebola vaccines that are nearly 100 percent effective; and cars that drive themselves.

After accounting for the contributions of labor and capital, economist Robert Solow calculated that nearly 90 percent of all improvements in living standards are due to technological progress. But we are handicapping ourselves with shoddy research practices and standards that waste tens of billions of dollars and send brilliant minds down scientific dead ends.

Publication Bias

There is no one single cause for the increase in nonreproducible findings in so many fields. One key problem is that the types of research most likely to make it from lab benches into leading scientific journals are those containing flashy never-before-reported results. Such findings are often too good to check. "All of the incentives are for researchers to write a good story—to provide journal editors with positive results, clean results, and novel results," notes the University of Virginia psychologist Brian Nosek. "This creates publication bias, and that is likely to be the central cause of the proliferation of false discoveries."

In 2014, the Cardiff University neuroscientist Christopher Chambers and his colleagues starkly outlined what they think is wrong. They noted that the gold standards for science are rigor, reproducibility, and transparency. But the academic career model has undermined those standards by instead emphasizing the production of striking results. "Within a culture that pressures scientists to produce rather than discover," they bleakly conclude, "the outcome is a biased and impoverished science in which most published results are either unconfirmed genuine discoveries or unchallenged fallacies."

As the editors of The Lancet put it in 2014, "Science is not done by paragons of virtue, but by individuals who are as prone to self-interest as anyone else." Such self-interest can take many forms, including seeking research grants and pursuing academic career advancement.

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  • jimw||

    The typo is most unfortunate: reproducibility rather than irreproducibility citing Freedman's report.

  • ||

    "HARKing is a similar practice in which scientists continue "hypothesizing after the results are known." If their initial hypothesis does not achieve statistical significance, the researchers retrospectively hunt through the data looking for some kind of positive result. The aim, again, is to report the "best" result they can extract from the data. In Nosek's words, they "tell a good story by reporting findings as if the research had been planned that way." This increases the chances of finding false positives and makes results seem stronger than they really are."

    Or they just go back and change the data...sorry, 'adjust'.

    I have been complaining for at least ten years that politics and money would corrupt science, but I have to say I did not see this coming. I didn't think it would be this bad or affect every field across the board.

  • Set Us Up The Chipper||

    Find it ironic that Bailey would talk about false discoveries, when he has been thoroughly duped by the CAGW establishment.

  • Scarecrow & WoodChipper Repair||

    Talk about cherry picking! Is that all you got out of this?

    And CAGW is not a black or white issue. Bailey is a lukewarmist, for instance. If you are one of the purist deniers, you are as biased and useless as one pf the purist warmers.

  • Spinach Chin||

    There are very few "purist deniers", but you are lumped together if you even suggest anything other than the propaganda that is shat out on a daily basis by progressive establishment.

    Bailey carries their water.

  • Set Us Up The Chipper||

    CAGW is pretty extreme by definition. The C does stand for CATASTROPHIC. I think Bailey is a little hotter than a luke warmer. My personal take is mostly along the lines of Curry (she used to be a warmer), McIntyre, Christy, etc. Humans affect the climate in many ways...deforestation (which is thankfully slowing way down), building human things, a small effect from pumping CO2 and aerosols into the atmosphere, etc.

    Ultimately, it's like Feynman said "It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong."

    So far, most if not all of the warmers predictions and the models they use fall into to the "doesn't agree with experiment" category. Their attitudes were exposed by Climategate, so how can one trust them now? Or to quote Merlin: "You betrayed the Duke, you stole his wife, you took his castle, now no-one trusts you"

    When, after the billions of government funding...way more than Big Oil ever paid any researcher, they come up with a theory that matches experiment, then I will listen.

  • ||

    Despite the current reproducibility crisis, Ioannidis says, "Science is, was, and will continue to be the best thing that has happened to human beings."

    If we ignore all the atrocities and failure, yeah, it's been great! Just like any one of those silly, irrational sky-daddy belief systems.

  • Tommy_Grand||

    The invention of the scientific process during the past two centuries is arguably humanity's greatest intellectual achievement.

    What about all the cool science humans did before 1816?

  • LynchPin1477||

    200 years doesn't go back far enough. I'd put it closer to the 16th and 17th centuries.

    If we are focusing on industrialization then 200 years seems closer.

  • ||

    Yeah, geometry, plumbing, and concrete are more ancient religion than any industrial science/technology anyway, fuck 'em.

    Running current through everything under the sun until you find the one thing that just happens to glow just right? Now that's Science!

  • danielhammond||

    Proving end points before disclosure that kind of real science or the statistical propaganda they have used for the last 200 years basically..........epidemiology is pure junk science anymore. Why did epidemiology drop using toxicologists to prove the claims...........because it couldn't be proven and that meant an end to the grant money gravy train so they invent statistics to advance the agenda,with so called linked to studies..........the world and everything in it has lost all faith in medicine or science much less trust in the courts or the law or the governments. Its just that bad just like the last time prohibition came around with the fanatical progressives.

  • danielhammond||

    In fact the magical mystical statiscal study has been used to drive radical political agendas for well over 100 years and was so bad even Mark Twain wrote about in 1894.......

    Mark Twain said it right over a hundred years ago:

    “The Moral Statistician.”
    Originally published in Sketches, Old and New, 1893

    "I don’t want any of your statistics; I took your whole batch and lit my pipe with it.

    I hate your kind of people. You are always ciphering out how much a man’s health is injured, and how much his intellect is impaired, and how many pitiful dollars and cents he wastes in the course of ninety-two years’ indulgence in the fatal practice of smoking; and in the equally fatal practice of drinking coffee; and in playing billiards occasionally; and in taking a glass of wine at dinner, etc. etc. And you are always figuring out how many women have been burned to death because of the dangerous fashion of wearing expansive hoops, etc. etc. You never see more than one side of the question.

  • danielhammond||

    You are blind to the fact that most old men in America smoke and drink coffee, although, according to your theory, they ought to have died young; and that hearty old Englishmen drink wine and survive it, and portly old Dutchmen both drink and smoke freely, and yet grow older and fatter all the time. And you never try to find out how much solid comfort, relaxation, and enjoyment a man derives from smoking in the course of a lifetime (which is worth ten times the money he would save by letting it alone), nor the appalling aggregate of happiness lost in a lifetime by your kind of people from not smoking. Of course you can save money by denying yourself all those little vicious enjoyments for fifty years; but then what can you do with it? What use can you put it to? Money can’t save your infinitesimal soul. All the use that money can be put to is to purchase comfort and enjoyment in this life; therefore, as you are an enemy to comfort and enjoyment where is the use of accumulating cash?

  • danielhammond||

    It won’t do for you to say that you can use it to better purpose in furnishing a good table, and in charities, and in supporting tract societies, because you know yourself that you people who have no petty vices are never known to give away a cent, and that you stint yourselves so in the matter of food that you are always feeble and hungry. And you never dare to laugh in the daytime for fear some poor wretch, seeing you in a good humor, will try to borrow a dollar of you; and in church you are always down on your knees, with your ears buried in the cushion, when the contribution-box comes around; and you never give the revenue officers a full statement of your income.

    Now you know all these things yourself, don’t you? Very well, then, what is the use of your stringing out your miserable lives to a lean and withered old age? What is the use of your saving money that is so utterly worthless to you? In a word, why don’t you go off somewhere and die, and not be always trying to seduce people into becoming as ornery and unlovable as you are yourselves, by your villainous “moral statistics”?"

  • Bra Ket||

    I don't buy that argument. Process is never a replacement for intelligence. Nor will it accomplish much in the absence of intelligence.

    As for 1816, the difference is just another example of technological determinism. As technology advances, science is enabled. And I think it's pretty clear that technology comes first, despite what all the central planners want to claim.

  • Jeffmo||

    Mysticism is what it is. Made up reality to satisfy someone's agenda. Notoriety has become more desired than reliability and intellectual honesty. Mysticism is taught to kids from the inception of their lives and throughout it without restraint. Thus,they lose their will and ability to accurately identify reality as they mature. Then they want to become scientists and researchers and doctors, etc.... They carry the propensity to distort reality with them into the professions and we get the situations that this article is highlighting.

  • danielhammond||

    You nailed it!

  • danielhammond||

    You nailed it!

  • Scarecrow & WoodChipper Repair||

    I wonder how much of this irreproducibility stems from so much science driven by supply of government funding, as opposed to demand of specific fields. Just as easy welfare, unemployment, and other government funds makes it easy to be lazy in looking for work, would not some similar effect be seen in doing research just because there is easy money, and no one actually cares about whether it is true or not? If funds were driven by demand instead, would not scientists concentrate on research which is actually useful, such that false results will be discovered much sooner?

    Or to put it another way, results which are interesting but generally useless won't be used elsewhere, and if they are false, no one will ever know. Whereas results which others actually have a use for will be replicated almost immediately, and often, merely from others trying to use them as the basis for further research.

  • Set Us Up The Chipper||

    Ridley's newest book talks about this in some detail. Effective science usually follows engineering and is mostly paid for by somebody besides the gov.

  • Scarecrow & WoodChipper Repair||

    That book is on one of my to-read shelves. Guess I better bump it up the queue.

  • LynchPin1477||

    Maybe. The medical results make me question that assertion, though. They may often be funded by government but they also are used widely. And most basic research in physics is still funded by the government. But then, maybe physics has just as much of a reproducibility problem...

    I suspect is has more to do the nature of the systems being studied (humans and their physiology or behavior).

  • dantheserene||

    The Lancet will forever be tainted for me by being the birthplace of the anti-vaxx movement. "Prestigious" is not a word I can associate with them.
    There must be better sources to cite.

  • Kratoklastes||

    If you are willing to lump any person who takes issue withe vaccination as practised into the cartoonish 'anti-vaxxer' trope, then you mark yourself out as someone of sub-standard cognition. It's the same level of stupidity that conflates criticism of the appalling models used to forecast climate (and they are absolutely abysmal) with being in favour of "let 'er rip" approaches to dumping ag runoff in aquifers.

    There is no inconsistency between believing that
    (1) vaccination - properly practised - works, and
    (2) vaccines produced by Big Pharma have side effects that more than offset their prophylactic efficacy (or conversely, by muffling the immune response, prevent longer-duration immunity - e.g., requiring an annual variant-specific flu vaccine instead of developing a natural immunity against a range of flu variants).

    And once you get government into the mix, it is almost P=1 that they (.gov) have chosen the side of the argument that harms public health (given their track record on everything from fluoride to sat-fat to weed) - which is why there's a fund for kiddies who have had adverse effects from vaccines, and it has paid out tens of millions of dollars to victims.

    People running the 'herd immunity' trope should vaccinate their kiddies - who will, therefore, be protected - and let those who prefer to allow innate immune responses, hold 'measles parties' (like the ones I went to as a kid).

  • LynchPin1477||

    The model he has in mind is the e-print distribution platform arXive. It is now common practice for physicists to post their pre-publication articles there to undergo scrutiny and assessment by other researchers.

    I personally do this, and have had some subtle mistakes pointed out before publication as a result. But there is still a cultural divide, and some people are adamantly opposed to posting to arxiv before peer review. Andalmost no one posts a really big discovery to the arxiv because they usually want to go for Nature or Science, and they don't want to get scooped.

  • danielhammond||

    Problem with Peer review is there aint no such thing anymore.............Its called rubber stamping your buddies work so he will do it for you.

  • danielhammond||

    Problem with Peer review is there aint no such thing anymore.............Its called rubber stamping your buddies work so he will do it for you.

  • AD-RtR/OS!||

    Oh, you mean like the trust we have in Climate Scientists like Phil Jones, and Michael Mann.
    What's to worry about, 97% say the science is settled.

  • danielhammond||

    Hows your hockey stick doing................I gather you didn't get your 2009 IPCC emails that were dumped on the world. Telling how the whole climate change deal is nothing but a massive hoax and extortion racket.

  • danielhammond||

    The rise of a pseudo-scientific links lobby

    Every day there seems to be a new study making a link between food, chemicals or lifestyle and ill-health. None of them has any link with reality.

    http://www.spiked-online.com/n.....6ibAzYo59A

  • danielhammond||

    Epidemiologists Vote to Keep Doing Junk Science

    Epidemiology Monitor (October 1997)

    An estimated 300 attendees a recent meeting of the American College of
    Epidemiology voted approximately 2 to 1 to keep doing junk science!

    Specifically, the attending epidemiologists voted against a motion
    proposed in an Oxford-style debate that “risk factor” epidemiology is
    placing the field of epidemiology at risk of losing its credibility.

    Risk factor epidemiology focuses on specific cause-and-effect
    relationships–like heavy coffee drinking increases heart attack risk. A
    different approach to epidemiology might take a broader
    perspective–placing heart attack risk in the context of more than just
    one risk factor, including social factors.

    Risk factor epidemiology is nothing more than a perpetual junk science machine.

    But as NIEHS epidemiologist Marilyn Tseng said “It’s hard to be an
    epidemiologist and vote that what most of us are doing is actually harmful
    to epidemiology.”

  • danielhammond||

    But who really cares about what they’re doing to epidemiology. I thought
    it was public health that mattered!

    we have seen the “SELECTIVE” blindness disease that
    Scientist have practiced over the past ten years. Seems the only color they
    see is GREEN BACKS, it’s a very infectious disease that has spread through
    the Scientific community with the same speed that any infectious disease
    would spread. And has affected the T(thinking) Cells as well as sight.

    Seems their eyes see only what their paid to see. To be honest, I feel
    after the Agent Orange Ranch Hand Study, and the Sl-utz and Nutz Implant
    Study, they have cast a dark shadow over their profession of being anything
    other than traveling professional witnesses for corporate hire with a lack
    of moral concern to their obligation of science and truth.

    The true “Risk Factor” is a question of ; will they ever be able to earn
    back the respect of their profession as an Oath to Science, instead of
    corporate paid witnesses with selective vision?
    Oh, if this seems way harsh, it’s nothing compared to the damage of peoples
    lives that selective blindness has caused!

  • danielhammond||

    Judge doesnt accept statistical studies as proof of LC causation!

    It was McTear V Imperial Tobacco. Here is the URL for both my summary and the Judge’s ‘opinion’ (aka ‘decision’):


    (2.14) Prof Sir Richard Doll, Mr Gareth Davies (CEO of ITL). Prof James Friend and
    Prof Gerad Hastings gave oral evidence at a meeting of the Health Committee in
    2000. This event was brought up during the present action as putative evidence that
    ITL had admitted that smoking caused various diseases. Although this section is quite
    long and detailed, I think that we can miss it out. Essentially, for various reasons, Doll
    said that ITL admitted it, but Davies said that ITL had only agreed that smoking might
    cause diseases, but ITL did not know. ITL did not contest the public health messages.
    (2.62) ITL then had the chance to tell the Judge about what it did when the suspicion
    arose of a connection between lung cancer and smoking. Researchers had attempted
    to cause lung cancer in animals from tobacco smoke, without success. It was right,
    therefore, for ITL to ‘withhold judgement’ as to whether or not tobacco smoke caused
    lung cancer.

  • danielhammond||

    [9.10] In any event, the pursuer has failed to prove individual causation.
    Epidemiology cannot be used to establish causation in any individual case, and the
    use of statistics applicable to the general population to determine the likelihood of
    causation in an individual is fallacious. Given that there are possible causes of lung
    cancer other than cigarette smoking, and given that lung cancer can occur in a nonsmoker,
    it is not possible to determine in any individual case whether but for an
    individual’s cigarette smoking he probably would not have contracted lung cancer
    (paras.[6.172] to [6.185]).
    [9.11] In any event there was no lack of reasonable care on the part of ITL at any
    point at which Mr McTear consumed their products, and the pursuer’s negligence
    case fails. There is no breach of a duty of care on the part of a manufacturer, if a
    consumer of the manufacturer’s product is harmed by the product, but the consumer
    knew of the product’s potential for causing harm prior to consumption of it. The
    individual is well enough served if he is given such information as a normally
    intelligent person would include in his assessment of how he wishes to conduct his
    life, thus putting him in the position of making an informed choice (paras.[7.167] to
    [7.181]).

  • Kratoklastes||

    Well, this is just evidence of judicial innumeracy (along with judicial second-rater-ness more generally: the Bench is a sinecure for second-rate advocates, with very very few exceptions).

    I re-heard an old joke the other day (on 'Better Call Saul'):

    Q: What do you call a lawyer with an IQ of 60?
    A: 'Your Honour'.

  • Brian||

    I never take published science as truth or fact beyond what it really *should* be claiming: an experiment was done, here's the result. You *might* be able to *infer* some truth from it, but, it could all be completely wrong, too.

    I treat published science as a call for reproduce checks, not as discoverers of truth, especially if it's a novel result.

    This can have some advantages. Sure, publication bias is bad, but, let's assume we were busy publishing every negative result, and then verifying every negative result. That would take a considerable amount of resources, too. Currently, publishing is biased towards novel results, and thus reproducibility studies are biased towards novel results, which is probably good enough.

    I don't think something is so much broken with science, as how it's interpreted and used. You don't have to live too long on this planet and pay attention for very long, either, before you realize that scientific studies are often quickly refuted.

  • danielhammond||

    Their all tied to influence peddling money. That's the problem be it government or big business or non profits with a prohibitionists eugenics agenda. All science today has only one purpose to push something for someone.

  • Jickerson||

    It never ceases to amaze me how often news organizations will cite some new study that just came out if they like its conclusions, all without caring that it hasn't been replicated or that there is not even any scientific consensus on the matter. But pretend it's true anyway, no matter how bad the study is; the matter is settled. Naturally, when the study is refuted, never report about that.

    This is particularly true with the social 'sciences', which are mostly made up of pure pseudoscience. They are very useful for authoritarians who want to control others.

  • Enemy of the State||

    Thank God the science surrounding catastrophic AGW is unblemished...

  • disillusioned||

    So... wait a minute. This story cites several news articles, which themselves cite singular academic studies to prove that most studies are false. That makes absolutely no sense at all. By their own logic it's oxymoronic and nonsensical. How can you use a study to prove that all studies are unreliable? Shouldn't we just dismiss the singular studies recently put out as flawed or potentially frauds if we're to believe them? How are we supposed to believe the meta-studies of the studies by their logic? Why should we believe that all these other studies are wrong just because one group tried one time to replicate them and couldn't? Why is it okay to listen to them but it's not okay to accept any other study at face value? Somewhere down the line we have to believe a study at face value at some point. If we don't, it'll never end, and we'll never be able to make technological advancements, make progress or do anything in life. All casting doubt on the science does is arm science denialists' cannons and through that hurt society and everyone else.

GET REASON MAGAZINE

Get Reason's print or digital edition before it’s posted online