Nullius in verba is the motto of one of the earliest scientific associations-the Royal Society, founded in 1663. Broadly translated, the phrase means "Don't take anybody's word for it."
You know how it's supposed to work: A scientist should ideally be able to do the same experiment as any other scientist and get similar results. As researchers check and recheck each other's findings, the sphere of knowledge expands. Replication is the path to scientific advancement.
Some 15 million researchers published more than 25 million scientific papers between 1996 and 2011. Among them were several casting doubt on the veracity and reliability of the rest—suggesting that even studies published in gold-standard journals by researchers from top-tier institutions are far more likely than anyone previously realized to be false, fudged, or flukey. The upshot is that many researchers have come to believe that science is badly battered, if not broken.
Everything We Know Is Wrong?
The Stanford statistician John Ioannidis sounded the alarm about our science crisis 10 years ago. "Most published research findings are false," Ioannidis boldly declared in a seminal 2005 PLOS Medicine article. What's worse, he found that in most fields of research, including biomedicine, genetics, and epidemiology, the research community has been terrible at weeding out the shoddy work largely due to perfunctory peer review and a paucity of attempts at experimental replication. Ioannidis showed, for instance, that about one-third of the results of highly cited original clinical research studies were shown to be wrong or exaggerated by subsequent research. "For many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias," he argued. Today, he says science is still wracked by the reproducibility problem: "In several fields, it is likely that most published research is still false."
Initially, some researchers argued that Ioannidis' claims were significantly overstated. "We don't think the system is broken and needs to be overhauled," declared New England Journal of Medicine editor Jeffrey Drazen in The Boston Globe in 2005. But his analyses have sparked a vast and ongoing reassessment of how science is done. Once other scientists started looking into the question, they found the same alarming trend everywhere.
In 2012, researchers at the pharmaceutical company Amgen reported in Nature that they were able to replicate the findings of only six out of 53 (11 percent) landmark published preclinical cancer studies. Preclinical studies test a drug, a procedure, or another medical treatment in animals as precursors to human trials. In 2011, researchers at Bayer Healthcare reported that they could not replicate 43 of the 67 published preclinical studies that the company had been relying on to develop cancer and cardiovascular treatments and diagnostics. Ioannidis estimates that "in biomedical sciences, non-replication rates that have been described range from more than 90 percent for observational associations (e.g., nutrient X causes cancer Y), to 75–90 percent for preclinical research (trying to find new drug targets)."
The mounting evidence that most scientific findings are false provoked a rash of worried headlines, including "How Science Is Broken" at Vox in 2015; "Why medical clinical trials are so wrong so often" in The Washington Post in 2015; "The Truth Is Many Scientific Studies Can't Be Trusted" at Business Insider in 2012; "Lies, Damned Lies, and Medical Science" in The Atlantic in 2010; and "Most Science Studies Appear to Be Tainted by Sloppy Analysis" in The Wall Street Journal in 2007.
In April 2015, the editor of the prestigious medical journal The Lancet reported that the participants in a recent conference believed "much of the scientific literature, perhaps half, may simply be untrue." In August, the journal Science reported that only one-third of 100 psychological studies published in three leading psychology journals could be adequately replicated. In October, a panel of senior researchers convened by the British Academy of Medical Sciences (BAMS) issued a major report on research reproducibility indicating that the false discovery rate in some areas of biomedicine could be as high as 69 percent.
If the rate of false positives is as high as feared, the Oxford researchers Ian Chalmers and Paul Glasziou suggest that as much as 85 percent of the resources devoted to biomedical research are being wasted. Globally, that amounts to about $200 billion every year.
In a June 2015 article for PLOS Biology, Leonard Freedman of the Global Biological Standards Institute and his colleagues note that published estimates for the reproducibility of preclinical research range from 51 percent to 89 percent. They estimate that at least half of all U.S. preclinical biomedical research funding—about $28 billion annually—is therefore squandered. As the colossal failure to replicate prominent cancer studies indicates, this wasted research results in treatments not developed and cures not found.
Venture capital firms now take it for granted, according to SciBX: Science-Business eXchange, that 50 percent of published academic studies cannot be replicated by industrial laboratories. Before investing in biomedical startups, they often hedge against "academic risk" by hiring contract research organizations to vet the science. This slows down the process of translating genuine discoveries into new products.
The invention of the scientific process during the past two centuries is arguably humanity's greatest intellectual achievement. Science and the technological progress it fosters have dramatically lengthened life spans, lessened the burdens of disease, reduced ignorance, and eased the hardships of work and daily life. We are living in a time of technological marvels, with advances like CRISPR gene-editing being used to bring back extinct mammoths; lithium-air batteries that store 10 times more energy than conventional lithium-ion batteries; mitochondrial transfers that create healthy babies who have three genetic parents; Ebola vaccines that are nearly 100 percent effective; and cars that drive themselves.
After accounting for the contributions of labor and capital, economist Robert Solow calculated that nearly 90 percent of all improvements in living standards are due to technological progress. But we are handicapping ourselves with shoddy research practices and standards that waste tens of billions of dollars and send brilliant minds down scientific dead ends.
There is no one single cause for the increase in nonreproducible findings in so many fields. One key problem is that the types of research most likely to make it from lab benches into leading scientific journals are those containing flashy never-before-reported results. Such findings are often too good to check. "All of the incentives are for researchers to write a good story—to provide journal editors with positive results, clean results, and novel results," notes the University of Virginia psychologist Brian Nosek. "This creates publication bias, and that is likely to be the central cause of the proliferation of false discoveries."
In 2014, the Cardiff University neuroscientist Christopher Chambers and his colleagues starkly outlined what they think is wrong. They noted that the gold standards for science are rigor, reproducibility, and transparency. But the academic career model has undermined those standards by instead emphasizing the production of striking results. "Within a culture that pressures scientists to produce rather than discover," they bleakly conclude, "the outcome is a biased and impoverished science in which most published results are either unconfirmed genuine discoveries or unchallenged fallacies."
As the editors of The Lancet put it in 2014, "Science is not done by paragons of virtue, but by individuals who are as prone to self-interest as anyone else." Such self-interest can take many forms, including seeking research grants and pursuing academic career advancement.