Are Most Scientific Results Bunk?
I recently enjoyed listening to Stanford University statistician John Ioannidis and University of Virginia psychologist Brian Nosek talk about how to make scientific evidence more reliable. Ioannides gained some well-deserved fame with his 2005 article in PLoS One, "Why Most Published Research Findings Are False," and Brian Nosek has just established the Center for Open Science which is offering its Open Science Framework that aims to improve the validity of scientific research.
In a recent article, "Evaluation of Very Large Treatment Effects of Medical Interventions," in the Journal of the American Medical Association, Ioannid1s and his colleagues combed through 85,000 medical interventions collected in the Cochrane Database of Systematic Reviews seeking to uncover highly effective treatments. What they found is that treatments that supposedly produce very large benefits (odds ratio greater than 5) were almost always found initially in small studies and that when they were replicated in larger studies the benefits became relatively modest. In the end only one treatment was found to provide a major benefit, e.g., supplying extracorporeal oxgyen to premature babies with severe respiratory failure. Last year, Nature reported the shocking finding that nine out of 10 preclinical peer-reviewed cancer research studies cannot be replicated.
Another big problem is the bias toward publishing positive results, while sticking negative results in the file drawer. In an interesting 2010 study published in PLoS One, University of Edinburgh researcher Daniele Fanelli found that as the science under consideration got "softer' the more positive results were reported. From the abstract:
This study analysed 2434 papers published in all disciplines and that declared to have tested a hypothesis. It was determined how many papers reported a "positive" (full or partial) or "negative" support for the tested hypothesis. If the hierarchy hypothesis is correct, then researchers in "softer" sciences should have fewer constraints to their conscious and unconscious biases, and therefore report more positive outcomes. Results confirmed the predictions at all levels considered: discipline, domain and methodology broadly defined. Controlling for observed differences between pure and applied disciplines, and between papers testing one or several hypotheses, the odds of reporting a positive result were around 5 times higher among papers in the disciplines of Psychology and Psychiatry and Economics and Business compared to Space Science, 2.3 times higher in the domain of social sciences compared to the physical sciences, and 3.4 times higher in studies applying behavioural and social methodologies on people compared to physical and chemical studies on non-biological material. In all comparisons, biological studies had intermediate values.
Last week, the Economist had a terrific article outlining the problems with lack of replicability and lax peer review in science, not least of which is that those problems mislead subsequent research efforts and is huge waste of money and talent. From the Economist:
The governments of the OECD, a club of mostly rich countries, spent $59 billion on biomedical research in 2012, nearly double the figure in 2000. One of the justifications for this is that basic-science results provided by governments form the basis for private drug-development work. If companies cannot rely on academic research, that reasoning breaks down. When an official at America's National Institutes of Health (NIH) reckons, despairingly, that researchers would find it hard to reproduce at least three-quarters of all published biomedical findings, the public part of the process seems to have failed.
The whole Economist article is well worth your time. Nosek's Open Science Framework project seems like a promising way to nudge researchers toward greater transparency and less data dredging. Through the system researchers can obtain "badges" for project pre-registration, open data, and open materials. Presumably these badges will help persuade journal editors to be more likely to publish such studies and thus encourage better research practices.
