The publishers Springer and IEEE are removing more than 120 papers from their subscription services after a French researcher discovered that the works were computer-generated nonsense.
The researcher is Cyril Labbé, a computer scientist who "has catalogued computer-generated papers that made it into more than 30 published conference proceedings between 2008 and 2013." According to Nature, he
developed a way to automatically detect manuscripts composed by a piece of software called SCIgen, which randomly combines strings of words to produce fake computer-science papers. SCIgen was invented in 2005 by researchers at the Massachusetts Institute of Technology (MIT) in Cambridge to prove that conferences would accept meaningless papers—and, as they put it, "to maximize amusement"….SCIgen is free to download and use, and it is unclear how many people have done so, or for what purposes. SCIgen's output has occasionally popped up at conferences, when researchers have submitted nonsense papers and then revealed the trick.
Labbé does not know why the papers were submitted—or even if the authors were aware of them. Most of the conferences took place in China, and most of the fake papers have authors with Chinese affiliations. Labbé has emailed editors and authors named in many of the papers and related conferences but received scant replies; one editor said that he did not work as a program chair at a particular conference, even though he was named as doing so, and another author claimed his paper was submitted on purpose to test out a conference, but did not respond on follow-up.
Retraction Watch notes that this story undercuts some of the conclusions people have drawn from a feature that Sciencepublished last fall. In that report, a researcher posing as a scholar at an imaginary African institute managed to publish nonsense papers in 304 open-access journals, a result touted in some quarters as showing a link between the open-access world and crappy quality control. (Open-access publications allow anyone to read their papers online, while conventional academic outlets charge high fees.) But Bohannon didn't submit his faux study to any traditional journals, so his results don't really allow you to compare the old and new models.
All of the outlets that Labbé exposed were subscription-based. The problem is evidently more extensive than some people thought.
Editor's Note: As of February 29, 2024, commenting privileges on reason.com
posts are limited to Reason Plus subscribers. Past commenters are grandfathered in for a temporary
period.
Subscribe
here to preserve your ability to comment. Your
Reason Plus subscription also gives you an ad-free version of reason.com, along with full access to the
digital
edition and archives of Reason magazine. We request that comments be civil and on-topic. We do
not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments
do not represent the views of reason.com or Reason Foundation. We reserve the right to delete any comment and
ban commenters for any reason at any time. Comments may only be edited within 5 minutes of posting. Report abuses.
Grand Moff Serious Man|2.25.14 @ 11:59AM|#
"Reich may be a fascist midget, but his writing is usually coherent."
To be honest, I rarely read more than the headline. When I have, it has been coherent. a
And consistent: *Every* column is about how the poor are getting rich more slowly than the rich are getting richer.
People worry about the emergence of an evil AI like the HAL 9000, and maybe we should, since a simple non intelligent algorithm can fool so many human experts.
Labb? is no stranger to fake studies. In April 2010, he used SCIgen to generate 102 fake papers by a fictional author called Ike Antkare [see pdf]. Labb? showed how easy it was to add these fake papers to the Google Scholar database, boosting Ike Antkare's h-index, a measure of published output, to 94 ? at the time, making Antkare the world's 21st most highly cited scientist.
Robin. "Then his bee this means something to he tree, so as head this bee. He can eat does? One day, Winnie-ther Robin. "It was a buzzing-noise, and a buzzing time the down a buzzing-noise, ander its making-noise little sure," said as he can this son for making to asked under likes 'under it." "Winnie-then he had the the had because means honey? Buzzing. You does? One door in to and, and bee that buzzing-noise. "Then himself: "And a loud began the forest, a last of the had topher Robin. "Now of the
It sounds like these were mostly submitted as conference proceedings or something similar. Those are not usually subjected to the same level of scrutiny as papers published in peer reviewed journals, and for exactly that reason the results are not usually as trusted (at least that is the case in the physical sciences). Maybe that should change. Until it does, studies like this one will highlight need to read things with a critical eye, which is a good thing. But it is important to make the distinction between publications subjected to peer review, and those that aren't.
It's not like fake articles haven't made it into peer-reviewed journals before. I suspect randomly generated articles wouldn't pass muster, just because someone on the editorial staff or in the review process would notice, but there's some crap out there that's "legitimately" published.
Well, Cold Fusion was debunked very quickly. By people who reviewed his experiment.
the Human embryo thing was more of a press stunt and had little to do with scientific journals if my memory serves. And I don't remember the Bogdanov affair. Did it involve hot chicks?
To be fair, journal articles do get some leeway in the harder sciences if they're in fields on the edge. The difference is that they usually have to temper their conclusions and are subject to others trying to replicate their results (or test their theories). I somehow doubt those things are happening as much in the softer sciences.
Oh absolutely, the peer reviewed process is certainly not flawless. But as you said, it takes a much more deliberate falsification of data, which, because of the effort, probably (hopefully) happens a lot less.
Full Disclosure, I am an officer in one of the IEEE's member societies.
The IEEE sponsors over 1300 conferences each year, and the average number of papers in an IEEE-sponsored conferences is over 200, so on average the IEEE publishes well over a quarter million conference papers each year. That is only conference papers, and does not consider the tens of thousands of journal and magazine articles also published by the IEEE. Having 120 retracted as gibberish is actually pretty darn good. And who's to say that the computer is actually worse than some human authors at writing papers?
Any system of review will err on letting in papers that should not be published and/or rejecting papers that should be published. This article seems to take the very non-libertarian position that the former is somehow less desirable than the latter. When judging papers myself (I both review and edit for the IEEE), I bias my evaluations in favor of free scientific speech and allowing authors to defend themselves, even if their position is undefensible.
Mamboo dogface to the banana patch. E plabnista!
http://www.anonstudiespublishedatconference.de/
Automated Sokal?
Fantastic.
Are they sure they weren't just Thomas Friedman op-eds?
R. Reich disputes your claim!
Reich may be a fascist midget, but his writing is usually coherent.
Unlike Friedman, Marcotte, any number of social justice/socialsit bloggers that get reposted here for our amusement/horror.
Grand Moff Serious Man|2.25.14 @ 11:59AM|#
"Reich may be a fascist midget, but his writing is usually coherent."
To be honest, I rarely read more than the headline. When I have, it has been coherent. a
And consistent: *Every* column is about how the poor are getting rich more slowly than the rich are getting richer.
By "coherent" you mean "he uses words."
So are transfats good again?
Not unless you like chubby chicks?
People worry about the emergence of an evil AI like the HAL 9000, and maybe we should, since a simple non intelligent algorithm can fool so many human experts.
Who says we are not already being powned by some AI?
I don't know about you motherfuckers, but I'm off to build up my CV.
With this, I'll make dean in a year!
If you need in-house counsel...
Is computer-generated nonsense worse or better than human-generated nonsense?
Better since you can't expect a computer to understand the "garbage in, garbage out" principle.
But just look at how many writers absorb absurdities without question and then reproduce it.
But is there consensus?
I see what you did there...
This really calls into question some pleadings I received the other day.
only the other day?
I managed to get out of mass tort litigation with my abilities to read and write barely intact.
Better, since the computer is morally neutral and assholes like Krugman are actively evil.
Well, lazily evil, wouldn't you say?
When it comes to philosophy, probably better.
Most of the comments on this blog are computer generated nonsense.
Robin. "Then his bee this means something to he tree, so as head this bee. He can eat does? One day, Winnie-ther Robin. "It was a buzzing-noise, and a buzzing time the down a buzzing-noise, ander its making-noise little sure," said as he can this son for making to asked under likes 'under it." "Winnie-then he had the the had because means honey? Buzzing. You does? One door in to and, and bee that buzzing-noise. "Then himself: "And a loud began the forest, a last of the had topher Robin. "Now of the
Cinnamon Toast - Toast!!! With penguins in tow...slap sheep upon brown cloud feathers.
The origins of settled science found!
Not to mention the source of Krugman's articles.
Krugman uses a lesser known program called KEYNSgen to write his articles.
http://apps.pdos.lcs.mit.edu/s.....irely.html
/removes glasses. Adjusts ascot. Locks jaw.
The transistor and Internet QoS, while confirmed in theory, have not until recently been considered appropriate.
BZZT!!
*** tears off Rufus's ascot ***
He lost me here:
Unified Bayesian technology have led to many compelling advances, including journaling file systems and IPv7
I think you lose your whistle for that.
But gains a motarboard
mortarboard
Funny Academic hat!
Whatever.
Finally the origin of the so-called Affordable Care Act (a.k.a. Obamacare) has been revealed!
Say, that could explain a lot.
Pass it before we know what's in it?
How could they tell?
So...I'm assuming there's a Palin's Buttplug version of this software.
It sounds like these were mostly submitted as conference proceedings or something similar. Those are not usually subjected to the same level of scrutiny as papers published in peer reviewed journals, and for exactly that reason the results are not usually as trusted (at least that is the case in the physical sciences). Maybe that should change. Until it does, studies like this one will highlight need to read things with a critical eye, which is a good thing. But it is important to make the distinction between publications subjected to peer review, and those that aren't.
It's not like fake articles haven't made it into peer-reviewed journals before. I suspect randomly generated articles wouldn't pass muster, just because someone on the editorial staff or in the review process would notice, but there's some crap out there that's "legitimately" published.
To words:
Sokal hoax.
Right, of course he comes to mind first. Of course, getting something across like that in the social sciences doesn't seem too difficult, does it?
getting something across like that in the social sciences doesn't seem too difficult, does it?
Spoken like a true Glibertarian!
Without Social Sciences, we wouldn't have all those degree programs which end in the word "Studies"!
Please. Cold fusion, that Korean guy who claimed to clone a bunch of human embryos, the Bogdanov affair.
No one's hands are clean.
Well, Cold Fusion was debunked very quickly. By people who reviewed his experiment.
the Human embryo thing was more of a press stunt and had little to do with scientific journals if my memory serves. And I don't remember the Bogdanov affair. Did it involve hot chicks?
To be fair, journal articles do get some leeway in the harder sciences if they're in fields on the edge. The difference is that they usually have to temper their conclusions and are subject to others trying to replicate their results (or test their theories). I somehow doubt those things are happening as much in the softer sciences.
Oh absolutely, the peer reviewed process is certainly not flawless. But as you said, it takes a much more deliberate falsification of data, which, because of the effort, probably (hopefully) happens a lot less.
Congress has been using LEGgen for decades. Why would this be a surprise?
Oh heck yeah dude lets roll with it.
http://www.Anon-VPN.com
Hey, I know why they did it! And why they were all Chinese!
Full Disclosure, I am an officer in one of the IEEE's member societies.
The IEEE sponsors over 1300 conferences each year, and the average number of papers in an IEEE-sponsored conferences is over 200, so on average the IEEE publishes well over a quarter million conference papers each year. That is only conference papers, and does not consider the tens of thousands of journal and magazine articles also published by the IEEE. Having 120 retracted as gibberish is actually pretty darn good. And who's to say that the computer is actually worse than some human authors at writing papers?
Any system of review will err on letting in papers that should not be published and/or rejecting papers that should be published. This article seems to take the very non-libertarian position that the former is somehow less desirable than the latter. When judging papers myself (I both review and edit for the IEEE), I bias my evaluations in favor of free scientific speech and allowing authors to defend themselves, even if their position is undefensible.