Shoring Up the Mantra of Science: Take Nobody's Word for It

The cited mantra is a general translation of "Nullius in verba," the motto of the British Royal Society, one of the world's first scientific organizations. Real science does not credit arguments from authority, but accepts the results from experiment and demonstration. The idea is that other researchers would check each others results to see if they could be reproduced. In the modern world, there's a lot less experimental replication and the result is lots of unreproduced experimental results are strewn throughout the scientific literature.

Earlier this year, two cancer researchers reported that that nine out of 10 preclinical peer-reviewed cancer research studies cannot be reproduced. As I explained in my column, "Can Most Cancer Research Be Trusted?":

The academic system encourages the publication of a lot of junk research, and former vice president for oncology research at the pharmaceutical company Amgen Glenn Begley and M.D. Anderson Cancer Center researcher Lee Ellis agree. “To obtain funding, a job, promotion or tenure, researchers need a strong publication record, often including a first-authored high-impact publication,” they note. And journal editors and grant reviewers make it worse by pushing researchers to produce “a scientific finding that is simple, clear and complete—a ‘perfect’ story.” This pressure induces some researchers massage data to fit an underlying hypothesis or even suppress negative data that contradicts the favored hypothesis. In addition, peer review is broken. If an article is rejected by one journal, very often researchers will ignore the comments of reviewers, slap on another cover letter and submit to another journal. The publication process becomes a lottery; not a way to filter out misinformation.

The company Science Exchange has proposed its "Reproducibility Initiative" as an innovative way to fix this problem at the heart of experimental science. As Science Daily reports:

Scientists who want to validate their findings will be able to apply to the initiative, which will choose a lab to redo the study and determine whether the results match.

The project sprang from the growing realization that the scientific literature - from social psychology to basic cancer biology - is riddled with false findings and erroneous conclusions, raising questions about whether such studies can be trusted. Not only are erroneous studies a waste of money, often taxpayers', but they also can cause companies to misspend time and resources as they try to invent drugs based on false discoveries.

"‘Published' and ‘true' are not synonyms," said Brian Nosek, a psychology professor at the University of Virginia in Charlottesville and a member of the initiative's advisory board....

The initiative's 10-member board of prominent scientists will match investigators with a lab qualified to test their results, said Elizabeth Iorns, Science Exchange's co-founder and chief executive officer. The original lab would pay the second for its work. How much depends on the experiment's complexity and the cost of study materials, but should not exceed 20 percent of the original research study's costs. Iorns hopes government and private funding agencies will eventually fund replication to improve the integrity of scientific literature.

The two labs would jointly write a paper, to be published in the journal PLoS One, describing the outcome. Science Exchange will issue a certificate if the original result is confirmed.

Here's hoping that lots of researchers will take advantage of this new initiative. For more background check out epidemiologist John Ioannides' 2005 classic article, "Why Most Published Research Findings are False" at PLoS Medicine.

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of Reason.com or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  • Fate||

    This would be truly awesome.

  • Brutus||

    And if the hard sciences like genetics and biotech are seeing this degree of fraud and phony findings, just imagine the quality of the research being churned out by the soft "sciences."

  • ||

    Yes, or even the medium sciences like health research (ranging from biochem to dieticians). How's that lipid hypothesis working out?

  • Cliché Bandit||

    Now THIS is true science. Chupacabra solved, Steve Smith still a mystery.

  • Brutus||

    I miss Art Bell.

  • Tman||

    That's why I love observing our advances in astrophysics so much. There is no room for bias or needing peer-acceptance. It either works or it doesn't, and the results are painfully obvious for everyone to see.

    The Curiosity Rover landing is easily one of the most advanced engineering feats ever achieved by mankind and it's a result of dedicated volunteers who committed themselves to pushing the limits of what they knew about the science involved. When you are attempting to land a 1 ton spacecraft autonomously from 100 million miles away there is no room for "opinions".

  • rts||

    it's a result of dedicated volunteers

    Volunteers, eh?

  • Tman||

    Well, yeah. Jpl/NASA/CAlTech, they all work there voluntarily. Sure they get paid, but it's different from other government run space programs in the sense that they aren't told that they will be working there.

    I suppose the pay part means they really aren't "volunteers" in one sense of the word, although the literal definition is "A person who freely offers to take part in an enterprise or undertake a task."

  • rts||

    The best kind of correct!

  • o3||

    NASA's robitics have rendered manned space flight obsolete...notwithstanding fat 1%ers playing flash gordon.

  • ||

    This interesting thing about astrophysics is that it is a purely observational science. Even things like Hawking radiation have yet to be verified experimentally. What I think is interesting about this is that astrophysics is considered a hard science but it doesn't follow people's idea of the Scientific Method with a capital 'S'. Performing an experiment with a null hypothesis, isolating variables, etc. Astrophysics is really about forming a hypothesis and then trying to observe something to confirm it.

  • Aresen||

    Actually, in astronomy and astrophysics it is generally the other way around.

    The astronomers observe something they haven't seen before, go WTF?. Then the astrophysicists try to figure out what it is.

    The astrophysicists propose models and the astronomers look for corroborating or exclusionary evidence.

    eg: Jocelyn Bell's observation of what later turned out to be neutron stars and the Vela satellite gamma ray burst observations.

    Both of these discoveries generated numerous hypotheses. The successful were later confirmed by other observations while the unsuccessful hypotheses were excluded.

  • Old Mexican||

    Re: Tman,

    The Curiosity Rover landing is easily one of the most advanced engineering feats ever achieved by mankind and it's a result of dedicated volunteers who committed themselves to pushing the limits of what they knew about the science involved.

    Arguably the two Viking landers were the most advanced engineering feats ever achieved when it comes to Mars, considering a) the landers required precise landing approaches; b)they already carried their own experiment labs inside; and c) this was done with computers as powerful as a child's V-Tech.

  • Tman||

    I disagree Old Mex, purely based on the fact that the sky crane used to lower Curiosity from the rocket boosters in mid-air was all done autonomously.

    The others simply landed, no big whoop.

  • Astra||

    That's why I love observing our advances in astrophysics so much. There is no room for bias or needing peer-acceptance. It either works or it doesn't, and the results are painfully obvious for everyone to see.

    Hah. Hah hah hah.

    Astrophysics has less trouble with inaccuracy in published papers than biomedical science, but the idea that there is not room for bias or peer-pressure is laughable. We are currently going through a funding crisis and the lemming-like rush to follow the "hot topics" is painful to watch.

    I am an astrophysicist who also builds hardware for ground-based and space missions. If you saw how the sausage is really made, you would be amazed that we ever launch anything successfully at all. I often am.

  • ||

    I've often wondered about this. Given the genetic and environmental variations between individuals, what if it simply isn't possible to reproduce some experiments exactly?

    What if the exact same individual, based on which brand of organge juice or urine (if he's a weirdo) he drank that morning, has a slightly different reaction to the exact same experiment? Does that invalidate both findings?

    I wonder if there are experiments, especially dealing with biology, that are both true and nonreplicatable.

  • Brutus||

    I don't think it invalidates the findings, since discovering that certain genetic experiements cannot be reproduced is itself a finding.

    The important thing to remember is that the only way an experiement is a total failure is if it doesn't tell you something. If A doesn't solve the problem, then you know not to try A again and to move on to B.

  • ||

    If that were the case, JJ, then the subject of the experiment would be truly unique. Which is mostly impossible; remember, their genes come from somewhere: their parents, so their parents have to have them too.

    Basically you are positing this situation: man drinks milk, has bad reaction. To reproduce this, they give another man milk, and he isn't lactose intolerant, therefore has no bad reaction.

    Yes, that can happen. But the pools of reaction are so wide that if the study is structured properly (not just one subject, but many, and so on), this will be taken into account.

    What you're concerned about should always be handled by a properly structured experiment.

  • tarran||

    This is yet another example of the vacuous comments packed with irreverent, obscure and irrelevant cultural references that you typically write, Episiarch, that are ruining Hit and Run.

  • SugarFree||

    He really is just the worst. He's The Britta of this place.

  • JW||

    I've always felt he was more the Balki of HampersandR.

  • SugarFree||

    "Don't be ridi-cool-us."

  • ||

    Balki?!? I don't even rate a Newman?!?

  • ||

    At least Gillian Jacobs is hot.

  • ||

    Well well well, if it isn't our resident troll and griefer, Episicunt. Slither out from under your rock to make a snide comment about Star Trek and pizza?

    You make me sick.

    And that's a reaction which I've found easily replicatable.

  • ||

    Well, that proves it then, does it not?

  • ||

    I dunno, I'm going to have to check with Lew Rockwell on this one. It might be a conspiracy to simply make it appear to be proof.

  • ||

    I wonder if there are experiments, especially dealing with biology, that are both true and nonreplicatable.

    Or very difficult to replicate, is there a difference? Given the ability to isolate all the variables in a biological organism (environmental, genetic, epigenetic) the experiment might be replicable, but that's pretty far away at this point for many hypothesis. Not to mention the fact that the many of the people doing this research are really mediocre.

  • WTF||

    Real science does not credit arguments from authority

    Or consensus, either.

  • DJF||

    “”””One can judge from experiment, or one can blindly accept authority. To the scientific mind, experimental proof is all important and theory is merely a convenience in description, to be junked when it no longer fits. To the academic mind, authority is everything and facts are junked when they do not fit theory laid down by authority.

    Robert A. Heinlein”””

  • R C Dean||

    In the modern world, there's a lot less experimental replication and the result is lots of unreproduced experimental results are strewn throughout the scientific literature.

    I wonder if that's a creation of grant-driven research funding. Who's going to issue a grant to replicate someone else's work?

  • ||

    So what you're saying is, the gov't needs to step in and fund all research!

    Market failure!

  • Cliché Bandit||

    And this has been a point of discussion for a while. Some have suggested more technical solutions like using particular formats to allow for automated x-refference and source checking and what not. And git type science platform allowing "branching" etc. Clearly a programmers paradigm but still a neat idea.This was a decent article in Ars.

  • tarran||

    All experiments in psychology are not of this [cargo cult] type, however. For example there have been many experiments running rats through all kinds of mazes, and so on — with little clear result. But in 1937 a man named Young did a very interesting one. He had a long corridor with doors all along one side where the rats came in, and doors along the other side where the food was. He wanted to see if he could train rats to go to the third door down from wherever he started them off. No. The rats went immediately to the door where the food had been the time before.
    The question was, how did the rats know, because the corridor was so beautifully built and so uniform, that this was the same door as before? Obviously there was something about the door that was different from the other doors. So he painted the doors very carefully, arranging the textures on the faces of the doors exactly the same. Still the rats could tell. Then he thought maybe they were smelling the food, so he used chemicals to change the smell after each run. Still the rats could tell. Then he realized the rats might be able to tell by seeing the lights and the arrangement in the laboratory like any commonsense person. So he covered the corridor, and still the rats could tell.
  • tarran||

    He finally found that they could tell by the way the floor sounded when they ran over it. And he could only fix that by putting his corridor in sand. So he covered one after another of all possible clues and finally was able to fool the rats so that they had to learn to go to the third door. If he relaxed any of his conditions, the rats could tell.

    Now, from a scientific standpoint, that is an A-number-one experiment. That is the experiment that makes rat-running experiments sensible, because it uncovers the clues that the rat is really using — not what you think it's using. And that is the experiment that tells exactly what conditions you have to use in order to be careful and control everything in an experiment with rat-running.
    I looked into the subsequent history of this research. The next experiment, and the one after that, never referred to Mr. Young. They never used any of his criteria of putting the corridor on sand, or of being very careful. They just went right on running rats in the same old way, and paid no attention to the great discoveries of Mr. Young, and his papers are not referred to, because he didn't discover anything about rats. In fact, he discovered all the things you have to do to discover something about rats. But not paying attention to experiments like that is a characteristic of cargo cult science.

    Richard Feynman: Cargo Cult Science

  • Brutus||

    Great man.

  • robc||

    Im interested to see which lab NASA pays to launch a series of satellites and take temperature readings. Especially since they would have to fund the time machine research too.

  • Old Mexican||

    "'Published' and 'true' are not synonyms," said Brian Nosek, a psychology professor at the University of Virginia in Charlottesville and a member of the initiative's advisory board


    Denier!!!!!!!

  • Alohi76||

    One reason research isn't done to replicate results: research is expensive. People/companies aren't interested in funding something that's already been done; journals aren't interested in publishing it. Scientists have been aware of this type of bias for decades.

  • Russell||

    Note to the reader :

    You're not obliged to swear by Ron's translation of any of Horace's words

GET REASON MAGAZINE

Get Reason's print or digital edition before it’s posted online

  • Progressive Puritans: From e-cigs to sex classifieds, the once transgressive left wants to criminalize fun.
  • Port Authoritarians: Chris Christie’s Bridgegate scandal
  • The Menace of Secret Government: Obama’s proposed intelligence reforms don’t safeguard civil liberties

SUBSCRIBE

advertisement