Don't Trust Citations
Andy Guess of Inside Higher Ed writes up a recent study that seeks to quantify errors in citations in scholarly papers. The results are more than a bit disturbing:
As it turns out, scholars have already done some work quantifying problem citations, divided into two categories, "incorrect references" and "quotation errors." The authors of the paper, J. Scott Armstrong of the University of Pennsylvania's Wharton School and Malcolm Wright of the Ehrenberg-Bass Institute at the University of South Australia, Adelaide, write of the former type, "This problem has been extensively studied in the health literature … 31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located."
More serious than such botched references are articles that incorrectly quote a cited paper or, as the authors put it, "misreport findings." For example, in the same study of health literature3, they write, "authors' descriptions of previous studies in public health journals differed from the original copy in 30 percent of references; half of these descriptions were unrelated to the quoting authors' contentions."
Editor's Note: As of February 29, 2024, commenting privileges on reason.com posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary period. Subscribe here to preserve your ability to comment. Your Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the digital edition and archives of Reason magazine. We request that comments be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Please
to post comments
How many incorrect citations did this study have?
A better question, Epi, is how many times did they misreport findings? That seems like a far more problematic issue.
See every citation made by John Walters or anyone form the DEA
Great study! I'll cite it the next time I'm showing my students that most studies are reliable.
What?
Universal college education will solve this problem.
It will not make it so that more people who haven't the slightest respect for academic integrity are able to publish influential studies full of preventable inaccuracies.
I've noticed this more and more in the journals I typically read. Authors like to cite, sometimes incorrectly, other studies just to please friends or co-authors. And many times it's just to get more references for a paper that is new. But I don't know much about how medical journal articles are put together.
31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located."
So any paper with more than three references is likely total bullshit?
Please critique any fallacy or error I employed in making that statement.
I HATE citations. One of the reasons I avoid direct statements.
See? And people think I am kidding when I say every citation I have gotten from a cop was false!
So, the bottom line here is that scholars have adopted the standards of contemporary journalism, right? Anyway, that's what I'll misquote the article as saying.
I hereby dub this the "Carmona Syndrome".
Blogs like Hit and Run are much better at links than citations in scholarly papers. If a link is bad or doesn't say what you claim, we will pwn you within 5 minutes. (citation needed)
Clicking on a link is a hell of a lot easier than going to the library and looking up a journal article.
I read the original article, and it clearly states that almost all academic citations are correct.
Don't believe me? Just check the original article. Besides, I'm an academic--I must know what I'm talking about.
"Without education, we are in a horrible and deadly danger of taking educated people seriously." --G. K. Chesterton [citation needed]
"Don't trust citations"
From careful examination, your title is not quite complete.
31 percent of the references in public health journals contained errors
As we see in the money quote, "Don't Trust Citations over 30" would be correct.
I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?
I wonder how this applies to legal opinions. Most lawyers, I think, use bluebook. Unless you're one of these tools.
Seventy-three per cent of all statistics citations are pulled out of....
thin air.
I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?
Is it that hard to think of fields that have been around for less than a century?
Dave B,
Is it that hard to think of fields that have been around for less than a century?
Yes, but I don't think he is looking for Climatology.
I might add (without reading the article) that this sounds like just one field. It's still pretty sad, but clinical or basic science research (or other social sciences) might have a far better citation record.
So any paper with more than three references is likely total bullshit?
By my math:
If a given citation has about a 2 in 3 chance of being accurate, then a mere two given citations have a 4 in 9 chance of being accurate. Once we cross the 1.9 citations per paper threshold (that's log 50% base 69%), more often than not we're reading something misleading. 23 citations (log 50% base 97%) and we get into likely false claim territory.
The inclination toward BS and filler should probably be intuitive, though, for anyone who's ever written a research paper.
Outta Cite!
How many people didn't get that reference?
I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?
UFOlogy! Someone call in to the Art Bell show to get a ruling.
Seventy-three per cent of all statistics citations are pulled out of....
thin air.
45 percent of all people know that.
I read the original article, and it clearly states that almost all academic citations are correct.
Don't believe me? Just check the original article. Besides, I'm an academic--I must know what I'm talking about. thoreau
I'm going to quote that in support of my next blog post, except it's a bit too long. The edited version will read:
I read it. Almost correct. Believe me? I'm an academic. -- Thor
Why are you getting snarky with me, DAR? Back off, man. I'm a scientist!
I wonder how this applies to legal opinions.
I suspect the first-level accuracy of legal work (can you find the source) is pretty high - its all brutally peer-reviewed by adversaries.
I suspect the second-level accuracy of legal work (does the source say what you claim) is, umm, open to debate. Which is the point, really, of adversarial/persuasive documents.
I'm curious as to the frequency of Ward Churchill-style citations to articles ghost-written by the author. That is REALLY the way to go!
Jon,
Wouldn't that be "spirit-written"? You know, since Mr. C was made an honorary Injun and all?
Why are you getting snarky with me, DAR? Back off, man. I'm a scientist!
But, but, thoreau! It's precisely because you're a scientist that I'm going to misquote you to add a sciency luster of truthish authority to my writing! Hey, can I borrow one of those lab coats? That would really make me look like I knew what I was talking about!
D.A. Ridgely,
Why does your proposal bring back horrid visions of Michael Dukakas riding in an M-1 tank?
You got my hopes up for a bit, but then I checked the SETI website. Their funders include the NASA Ames Research Center, NASA Headquarters,National Science Foundation, Department of Energy and US Geological Survey. Ah well, I guess I'll have to go into statistics instead science research to preserve my Libertarian views. Mathematics is the one thing I can afford to self-fund. :-\
jtuf,
No no no! Not those guys, they are just a front group for the man.
Try Erik von Danaken and his cronies. Although, they may be getting a few kroners from abroad.
There are those who can show you how to be a REAL scientist. Seek him my son.
Universal college education will solve this problem.
I agree, Reinmoose. The more exposure people have to PhDs, the less weight their opinions will carry.
Yea Kwix, make him do it the hard way instead of the way that "Men are from Mars" guy and his ex-wife did it.
man oh man, you can be brutal sometimes.
If the proof is right, you need not cite.
Seems like an easy problem to solve. Find some undergrad who wants course credit or a letter of recommendation to grad school to hang out at the office and follow the citations to make sure everything checks out. You don't even need to pay them.
On a related note, I've been impressed by the number of articles where the conclusion in the abstract contradicts the data in the article.
I was going to do a study of this issue, but why bother? Let's just say it is 37% of the time.
You may quote me.
Thanks! :->
I know lots about not very much. Recently a 500+ page dissertation that touched on my subject came my way. Everything it had to say about my subject was wrong is some way - direct quotes uncited, a quote cited to me that I didn't say, my name misspelled in the citation. I shudder to think about the potential accuracy of the rest of the dissertation. Someone got a PhD for it though.
Hm.
Seems like a good point. At 500+ pages, I'm sure I'd need someone to edit and proofread my work, too. And when I was an undergrad, I'd cherish an experience like that.*
*Assuming the paper was any good and that I'd get course credit.
Dr. Venture? I love that show!
Thanks to all who've brought this thread back to the beat generation
Seems like a good point