Reason Magazine

Site Search

Don't Trust Citations

Andy Guess of Inside Higher Ed writes up a recent study that seeks to quantify errors in citations in scholarly papers. The results are more than a bit disturbing:

As it turns out, scholars have already done some work quantifying problem citations, divided into two categories, "incorrect references" and "quotation errors." The authors of the paper, J. Scott Armstrong of the University of Pennsylvania's Wharton School and Malcolm Wright of the Ehrenberg-Bass Institute at the University of South Australia, Adelaide, write of the former type, "This problem has been extensively studied in the health literature ... 31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located."

More serious than such botched references are articles that incorrectly quote a cited paper or, as the authors put it, "misreport findings." For example, in the same study of health literature3, they write, "authors' descriptions of previous studies in public health journals differed from the original copy in 30 percent of references; half of these descriptions were unrelated to the quoting authors' contentions."

Whole story here.

Send this article to:

« New at Reason: Tim Cavanaugh… | Main | Why Does Pixar Hate America? »

Comments to "Don't Trust Citations":

Episiarch | July 10, 2008, 12:17pm | #

How many incorrect citations did this study have?

T | July 10, 2008, 12:19pm | #

A better question, Epi, is how many times did they misreport findings? That seems like a far more problematic issue.

Warren | July 10, 2008, 12:20pm | #

See every citation made by John Walters or anyone form the DEA

thoreau | July 10, 2008, 12:20pm | #

Great study! I'll cite it the next time I'm showing my students that most studies are reliable.

What?

Reinmoose | July 10, 2008, 12:20pm | #

Universal college education will solve this problem.
It will not make it so that more people who haven't the slightest respect for academic integrity are able to publish influential studies full of preventable inaccuracies.

Kyle | July 10, 2008, 12:21pm | #

I've noticed this more and more in the journals I typically read. Authors like to cite, sometimes incorrectly, other studies just to please friends or co-authors. And many times it's just to get more references for a paper that is new. But I don't know much about how medical journal articles are put together.

SIV | July 10, 2008, 12:25pm | #

31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located."

So any paper with more than three references is likely total bullshit?

Please critique any fallacy or error I employed in making that statement.

Naga Sadow | July 10, 2008, 12:26pm | #

I HATE citations. One of the reasons I avoid direct statements.

Guy Montag | July 10, 2008, 12:28pm | #

See? And people think I am kidding when I say every citation I have gotten from a cop was false!

D.A. Ridgely | July 10, 2008, 12:42pm | #

So, the bottom line here is that scholars have adopted the standards of contemporary journalism, right? Anyway, that's what I'll misquote the article as saying.

Kwix | July 10, 2008, 12:44pm | #

"authors' descriptions of previous studies in public health journals differed from the original copy in 30 percent of references; half of these descriptions were unrelated to the quoting authors' contentions."
I hereby dub this the "Carmona Syndrome".

tomhynes | July 10, 2008, 12:51pm | #

Blogs like Hit and Run are much better at links than citations in scholarly papers. If a link is bad or doesn't say what you claim, we will pwn you within 5 minutes. (citation needed)

Clicking on a link is a hell of a lot easier than going to the library and looking up a journal article.

thoreau | July 10, 2008, 12:54pm | #

I read the original article, and it clearly states that almost all academic citations are correct.

Don't believe me? Just check the original article. Besides, I'm an academic--I must know what I'm talking about.

ErĂ¿k Boston, J.D. | July 10, 2008, 12:58pm | #

"Without education, we are in a horrible and deadly danger of taking educated people seriously." --G. K. Chesterton [citation needed]

Guy Montag | July 10, 2008, 12:58pm | #

"Don't trust citations"

From careful examination, your title is not quite complete.

31 percent of the references in public health journals contained errors

As we see in the money quote, "Don't Trust Citations over 30" would be correct.

jtuf | July 10, 2008, 12:59pm | #

I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?

hotsauce | July 10, 2008, 1:02pm | #

I wonder how this applies to legal opinions. Most lawyers, I think, use bluebook. Unless you're one of these tools.

P Brooks | July 10, 2008, 1:03pm | #

Seventy-three per cent of all statistics citations are pulled out of....



thin air.

Dave B. | July 10, 2008, 1:06pm | #

I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?

Is it that hard to think of fields that have been around for less than a century?

Guy Montag | July 10, 2008, 1:10pm | #

Dave B,

Is it that hard to think of fields that have been around for less than a century?

Yes, but I don't think he is looking for Climatology.

J | July 10, 2008, 1:12pm | #

I might add (without reading the article) that this sounds like just one field. It's still pretty sad, but clinical or basic science research (or other social sciences) might have a far better citation record.

Amakudari | July 10, 2008, 1:13pm | #

So any paper with more than three references is likely total bullshit?

By my math:

If a given citation has about a 2 in 3 chance of being accurate, then a mere two given citations have a 4 in 9 chance of being accurate. Once we cross the 1.9 citations per paper threshold (that's log 50% base 69%), more often than not we're reading something misleading. 23 citations (log 50% base 97%) and we get into likely false claim territory.

The inclination toward BS and filler should probably be intuitive, though, for anyone who's ever written a research paper.

Art-P.O.G. | July 10, 2008, 1:15pm | #

Outta Cite!

How many people didn't get that reference?

Guy Montag | July 10, 2008, 1:16pm | #

I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?

UFOlogy! Someone call in to the Art Bell show to get a ruling.

Butts Wagner | July 10, 2008, 1:22pm | #

Seventy-three per cent of all statistics citations are pulled out of....



thin air.


45 percent of all people know that.

D.A. Ridgely | July 10, 2008, 1:25pm | #

I read the original article, and it clearly states that almost all academic citations are correct.

Don't believe me? Just check the original article. Besides, I'm an academic--I must know what I'm talking about.
thoreau

I'm going to quote that in support of my next blog post, except it's a bit too long. The edited version will read:

I read it. Almost correct. Believe me? I'm an academic. -- Thor

thoreau | July 10, 2008, 1:26pm | #

Why are you getting snarky with me, DAR? Back off, man. I'm a scientist!

R C Dean | July 10, 2008, 1:32pm | #

I wonder how this applies to legal opinions.

I suspect the first-level accuracy of legal work (can you find the source) is pretty high - its all brutally peer-reviewed by adversaries.

I suspect the second-level accuracy of legal work (does the source say what you claim) is, umm, open to debate. Which is the point, really, of adversarial/persuasive documents.

Jon | July 10, 2008, 1:37pm | #

I'm curious as to the frequency of Ward Churchill-style citations to articles ghost-written by the author. That is REALLY the way to go!

Guy Montag | July 10, 2008, 1:51pm | #

Jon,

Wouldn't that be "spirit-written"? You know, since Mr. C was made an honorary Injun and all?

D.A. Ridgely | July 10, 2008, 1:52pm | #

Why are you getting snarky with me, DAR? Back off, man. I'm a scientist!

But, but, thoreau! It's precisely because you're a scientist that I'm going to misquote you to add a sciency luster of truthish authority to my writing! Hey, can I borrow one of those lab coats? That would really make me look like I knew what I was talking about!

Guy Montag | July 10, 2008, 2:04pm | #

D.A. Ridgely,

Why does your proposal bring back horrid visions of Michael Dukakas riding in an M-1 tank?

jtuf | July 10, 2008, 2:05pm | #

Guy Montag | July 10, 2008, 1:16pm | #

I wonder how the results would differ in field that hasn't been entwined with the government for the past century. For example, let's look at ... um ... any suggestions?

UFOlogy! Someone call in to the Art Bell show to get a ruling.
You got my hopes up for a bit, but then I checked the SETI website. Their funders include the NASA Ames Research Center, NASA Headquarters,National Science Foundation, Department of Energy and US Geological Survey. Ah well, I guess I'll have to go into statistics instead science research to preserve my Libertarian views. Mathematics is the one thing I can afford to self-fund. :-

Guy Montag | July 10, 2008, 2:10pm | #

jtuf,

No no no! Not those guys, they are just a front group for the man.

Try Erik von Danaken and his cronies. Although, they may be getting a few kroners from abroad.

Kwix | July 10, 2008, 2:25pm | #

thoreau | July 10, 2008, 1:26pm | #
Why are you getting snarky with me, DAR? Back off, man. I'm a scientist!
There are those who can show you how to be a REAL scientist. Seek him my son.

Hugh Akston | July 10, 2008, 2:33pm | #

Universal college education will solve this problem.

I agree, Reinmoose. The more exposure people have to PhDs, the less weight their opinions will carry.

Guy Montag | July 10, 2008, 2:34pm | #

Yea Kwix, make him do it the hard way instead of the way that "Men are from Mars" guy and his ex-wife did it.

man oh man, you can be brutal sometimes.

Someone Who Doesn't Want to Lose His Job | July 10, 2008, 2:34pm | #

If the proof is right, you need not cite.

Travis | July 10, 2008, 3:37pm | #

Seems like an easy problem to solve. Find some undergrad who wants course credit or a letter of recommendation to grad school to hang out at the office and follow the citations to make sure everything checks out. You don't even need to pay them.

Slugger | July 10, 2008, 3:41pm | #

On a related note, I've been impressed by the number of articles where the conclusion in the abstract contradicts the data in the article.
I was going to do a study of this issue, but why bother? Let's just say it is 37% of the time.
You may quote me.

jtuf | July 10, 2008, 4:46pm | #

Guy Montag | July 10, 2008, 2:10pm | #

jtuf,

No no no! Not those guys, they are just a front group for the man.

Try Erik von Danaken and his cronies. Although, they may be getting a few kroners from abroad.
Thanks! :->

RP | July 10, 2008, 6:56pm | #

I know lots about not very much. Recently a 500+ page dissertation that touched on my subject came my way. Everything it had to say about my subject was wrong is some way - direct quotes uncited, a quote cited to me that I didn't say, my name misspelled in the citation. I shudder to think about the potential accuracy of the rest of the dissertation. Someone got a PhD for it though.

Art-P.O.G. | July 11, 2008, 2:51am | #

Recently a 500+ page dissertation that touched on my subject came my way. Everything it had to say about my subject was wrong is some way
Hm.
Seems like an easy problem to solve. Find some undergrad who wants course credit or a letter of recommendation to grad school to hang out at the office and follow the citations to make sure everything checks out. You don't even need to pay them.
Seems like a good point. At 500+ pages, I'm sure I'd need someone to edit and proofread my work, too. And when I was an undergrad, I'd cherish an experience like that.*

*Assuming the paper was any good and that I'd get course credit.

Art-P.O.G. | July 11, 2008, 2:52am | #

There are those who can show you how to be a REAL scientist.
Dr. Venture? I love that show!