Shoring Up the Mantra of Science: Take Nobody's Word for It

The cited mantra is a general translation of "Nullius in verba," the motto of the British Royal Society, one of the world's first scientific organizations. Real science does not credit arguments from authority, but accepts the results from experiment and demonstration. The idea is that other researchers would check each others results to see if they could be reproduced. In the modern world, there's a lot less experimental replication and the result is lots of unreproduced experimental results are strewn throughout the scientific literature.

Earlier this year, two cancer researchers reported that that nine out of 10 preclinical peer-reviewed cancer research studies cannot be reproduced. As I explained in my column, "Can Most Cancer Research Be Trusted?":

The academic system encourages the publication of a lot of junk research, and former vice president for oncology research at the pharmaceutical company Amgen Glenn Begley and M.D. Anderson Cancer Center researcher Lee Ellis agree. "To obtain funding, a job, promotion or tenure, researchers need a strong publication record, often including a first-authored high-impact publication," they note. And journal editors and grant reviewers make it worse by pushing researchers to produce "a scientific finding that is simple, clear and complete—a 'perfect' story." This pressure induces some researchers massage data to fit an underlying hypothesis or even suppress negative data that contradicts the favored hypothesis. In addition, peer review is broken. If an article is rejected by one journal, very often researchers will ignore the comments of reviewers, slap on another cover letter and submit to another journal. The publication process becomes a lottery; not a way to filter out misinformation.

The company Science Exchange has proposed its "Reproducibility Initiative" as an innovative way to fix this problem at the heart of experimental science. As Science Daily reports:

Scientists who want to validate their findings will be able to apply to the initiative, which will choose a lab to redo the study and determine whether the results match.

The project sprang from the growing realization that the scientific literature - from social psychology to basic cancer biology - is riddled with false findings and erroneous conclusions, raising questions about whether such studies can be trusted. Not only are erroneous studies a waste of money, often taxpayers', but they also can cause companies to misspend time and resources as they try to invent drugs based on false discoveries.

"'Published' and 'true' are not synonyms," said Brian Nosek, a psychology professor at the University of Virginia in Charlottesville and a member of the initiative's advisory board….

The initiative's 10-member board of prominent scientists will match investigators with a lab qualified to test their results, said Elizabeth Iorns, Science Exchange's co-founder and chief executive officer. The original lab would pay the second for its work. How much depends on the experiment's complexity and the cost of study materials, but should not exceed 20 percent of the original research study's costs. Iorns hopes government and private funding agencies will eventually fund replication to improve the integrity of scientific literature.

The two labs would jointly write a paper, to be published in the journal PLoS One, describing the outcome. Science Exchange will issue a certificate if the original result is confirmed.

Here's hoping that lots of researchers will take advantage of this new initiative. For more background check out epidemiologist John Ioannides' 2005 classic article, "Why Most Published Research Findings are False" at PLoS Medicine.