Media Criticism

Experts Fooled by Super Computer or Media Fooled by Bad Test?


The reports are impressive, maybe even a little spooky: On Saturday, a supercomputer finally fooled the experts, convincing them that they were chatting not with a machine but a real, live human. But more discerning tech experts say that the test was poorly administered and that the equipment is not as impressive as the media has painted it.

The Independent claims that a "super computer" with the fictitious identity of a 13-year-old Ukrainian boy named Eugene Goostman "is thought to be the first computer to pass the iconic" Turing Test. Goostman was designed by a Russian team and speaks broken English. Tech-centric sites like Ars Technica and Computerworld repeated the claim that Goostman is a "super computer."

Yahoo News explains what the Turing Test is:

The Turing Test measures a computer's intelligence level through conversations that take place between human and machine, and the machine is said to have passed if the human at the other end cannot tell that it is an artificial intelligence system that he/she is conversing with.

If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now.

Yahoo adds that this is "a landmark moment for artificial intelligence."

TechDirt's Mike Masnick has a point-by-point take down of the overhyped situation:

It's not a "super computer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot. Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59 percent of judges it was human (much higher than the 33 percent Eugene Goostman) claims. It "beat" the Turing test here by "gaming" the rules—by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses. The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway. As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right? The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction. Masnick also warns readers that Kevin Warwick, who organized the event, has over the years been behind several sensational headlines that don't reflect reality. 

Gawker's tech blog, io9, highlights similar issues, but adds, "In other words, this is far from the milestone it's been made out to be. That said, it is important, because it supports the idea that we have entered an era in which it will become increasingly difficult to discern chatbots from real humans."

Chat with Goostman here and see if you're convinced. 

NEXT: Ira Stoll on the New Face of Conservative Reform

Editor's Note: We invite comments and request that they be civil and on-topic. We do not moderate or assume any responsibility for comments, which are owned by the readers who post them. Comments do not represent the views of or Reason Foundation. We reserve the right to delete any comment for any reason at any time. Report abuses.

  1. I agree that the whole affair has been way overhyped, but:

    It’s not a “super computer,” it’s a chatbot. It’s a script made to mimic human conversation. There is no intelligence, artificial or not involved. It’s just a chatbot.

    doesn’t make a lick of sense. That something is a chatbot says nothing about what hardware it runs on. Nor does it say anything about its complexity or lack thereof.

    And does Masnick understand what “script” means in a computing context? It seems like he ought to, but the incredulous way he uses the term suggests he doesn’t.

    1. I took the statement in toto. It’s not Watson. It’s not even lexing/parsing the English or employing vast NLP libraries. It’s just picking out the keywords and doing a db lookup for a match.

  2. “If someone claimed to have created nuclear fusion or cured cancer, you’d wait for some peer review and repeat tests under other circumstances before buying it, right?”

    Yes. Sadly, I’d also predict it would be “Breaking News” at, above the fold on the first page of the Science section of the Times and commented favorably upon by Vice President Biden.

  3. It seems to me that using the “thirteen year old boy” gimmick is a way to limit even the questionable value of the Turing test. Can I claim that a potato has passed the Turing test if I put it in front of my keyboard and then invite people to chat with my comatose great uncle, and they come away thinking they were chatting to a comatose human instead of a perfectly healthy potato?

    1. That’s already been done, it’s called SIRI.

  4. “If someone claimed to have created nuclear fusion”

    1952 called and they want due credit.…..rogen-bomb

  5. Eugenuslav Goostmananov?

  6. The whole test oversimplifies the concept of artificial intelligence. A more appropriate test would focus on cognitive ability. Testing the capacity of a system to actually learn and apply concepts should first measure how it processes the information to see whether or not the process replicates human thought in some manner.

    Displaying the complexity needed for real intelligence is completely different than fooling a human into thinking they are chatting with a 13 year old kid who speaks in broken English.

Please to post comments

Comments are closed.