Experts Fooled by Super Computer or Media Fooled by Bad Test?

The reports are impressive, maybe even a little spooky: On Saturday, a supercomputer finally fooled the experts, convincing them that they were chatting not with a machine but a real, live human. But more discerning tech experts say that the test was poorly administered and that the equipment is not as impressive as the media has painted it.

The Independent claims that a "super computer" with the fictitious identity of a 13-year-old Ukrainian boy named Eugene Goostman "is thought to be the first computer to pass the iconic" Turing Test. Goostman was designed by a Russian team and speaks broken English. Tech-centric sites like Ars Technica and Computerworld repeated the claim that Goostman is a "super computer."

Yahoo News explains what the Turing Test is:

The Turing Test measures a computer's intelligence level through conversations that take place between human and machine, and the machine is said to have passed if the human at the other end cannot tell that it is an artificial intelligence system that he/she is conversing with.

If a computer is mistaken for a human more than 30% of the time during a series of five minute keyboard conversations it passes the test. No computer has ever achieved this, until now.

Yahoo adds that this is "a landmark moment for artificial intelligence."

TechDirt's Mike Masnick has a point-by-point take down of the overhyped situation:

It's not a "super computer," it's a chatbot. It's a script made to mimic human conversation. There is no intelligence, artificial or not involved. It's just a chatbot. Plenty of other chatbots have similarly claimed to have "passed" the Turing test in the past (often with higher ratings). Here's a story from three years ago about another bot, Cleverbot, "passing" the Turing Test by convincing 59 percent of judges it was human (much higher than the 33 percent Eugene Goostman) claims. It "beat" the Turing test here by "gaming" the rules -- by telling people the computer was a 13-year-old boy from Ukraine in order to mentally explain away odd responses. The "rules" of the Turing test always seem to change. Hell, Turing's original test was quite different anyway. As Chris Dixon points out, you don't get to run a single test with judges that you picked and declare you accomplished something. That's just not how it's done. If someone claimed to have created nuclear fusion or cured cancer, you'd wait for some peer review and repeat tests under other circumstances before buying it, right? The whole concept of the Turing Test itself is kind of a joke. While it's fun to think about, creating a chatbot that can fool humans is not really the same thing as creating artificial intelligence. Many in the AI world look on the Turing Test as a needless distraction. Masnick also warns readers that Kevin Warwick, who organized the event, has over the years been behind several sensational headlines that don't reflect reality. 

Gawker's tech blog, io9, highlights similar issues, but adds, "In other words, this is far from the milestone it's been made out to be. That said, it is important, because it supports the idea that we have entered an era in which it will become increasingly difficult to discern chatbots from real humans."

Chat with Goostman here and see if you're convinced.