Since its earliest days, Wired magazine has always had a genius for publicity. It lost its edge somewhat when Condé Nast took over two years ago and the new crew replaced Wired’s Bay Area techno-exuberance with the New York publishing formulas that eschew ideas in favor of celebrities.
But the hype machine came roaring back with the April cover story–a long, long, long think piece by the hip software genius Bill Joy, chief scientist at Sun Microsystems. (The article can be read at www.wired.com/wired/archive/8.04/joy.html.) Joy was the key designer of Berkeley Unix and the man behind Sun’s Java and Jini. He is a techno-celebrity of the first order.
But he’s not a geeky geek. With his stylish all-black outfits and narrow eyeglass frames, Joy looks more Hollywood than Silicon Valley. (He actually lives in Aspen.) And he’s as earnest and elitist as a Harvard professor. In short, he’s the perfect authority to book on TV. And he had the perfect message: Technological research must be stopped.
Specifically, Joy is worried, really worried–20,000 words and five months of writing worried–that 21st-century technologies threaten to make human beings extinct. The threats are intelligent robots, nanotechnology (the ability to build things on the atomic level), and genetic engineering. All of them, he acknowledges, offer wonderful advantages, but they are, in his view, simply too dangerous to develop. We should stop investigating these ideas, he argues, before they become uncontrollable realities.
"The new Pandora’s boxes of genetics, nanotechnology, and robotics are almost open. Ideas can’t be put back in a box; unlike uranium or plutonium, they don’t need to be mined or refined, and they can be freely copied," he writes. "Once they are out, they are out." So we’d better keep them safely unknown.
Joy’s article starts with the idea that highly intelligent robots might supercede human beings. This could happen because a) the robots are so great we become dependent on them and essentially bore ourselves to death (a scenario Joy snagged from the Unabomber manifesto) or b) the robots outcompete us for economic resources, so that we can’t afford enough food, water, land, energy, etc., to survive, just as placental mammals wiped out competing marsupials in the Americas (an idea from robotics researcher Hans Moravec).
There’s also the possibility, outlined by Ray Kurzweil in The Age of Spiritual Machines, that we might gradually merge with our machines, enhancing our intelligence or uploading our consciousness, and thus change the nature of humanity. Kurzweil and his book inspired Joy’s nightmares, but Joy doesn’t really explore Kurzweil’s scenario.
Instead, he moves quickly to a nearer-term threat–the possibility that someone might use genetic engineering to create a devastating plague. This scenario illustrates the fundamental problems of these new technologies. Unlike 20th-century weapons of mass destruction, notably nuclear bombs, these are hard to limit to a small cadre of government-approved scientists and engineers.
"For the first time," writes Joy, "these accidents and abuses are widely within the reach of individuals or small groups. They will not require large facilities or rare raw materials. Knowledge alone will enable the use of them." We will face "a surprising and terrible empowerment of extreme individuals." In the future, the Unabomber will be a molecular biologist. And, once created, these technologies will have the ability to reproduce and spread, to self-replicate beyond the control of their creators and threaten widespread destruction.
To avoid that prospect, Joy advocates "relinquishment: to limit development of the technologies that are too dangerous, by limiting our pursuit of certain kinds of knowledge." He would enforce that limit through "a verification regime similar to that for biological weapons, but on an unprecedented scale," combined with a scientific code of conduct that forbids such research. That’s the prescription that made TV bookers and newspaper editorialists sit up and take notice. A scientist was saying we should stop scientific research.
The technologies Joy fears do indeed have their dark side, and it does lie in their easy, decentralized production and subsequent self-replication. These technologies can be genuinely scary. But Joy’s policy prescriptions are breathtakingly naive, blinkered, and totalitarian. They are contradicted by his own arguments, and by the uncomfortable historical realities Joy skirts even as he invokes them. Though it strains to seem wise, Joy’s manifesto is at once childlike and childish.
Introducing Joy on Good Morning America, Charles Gibson explained the purpose of the article: "The contents of an article coming out this morning in Wired magazine are being compared to the 1939 letter that Albert Einstein wrote to President Roosevelt warning of the consequences of the atomic bomb. Today’s warning that new technology could mean the extermination of mankind. It comes from a respected scientist who was one of the key architects of the information age, and that’s why his warning is causing such a stir in The Washington Post and other newspapers."
The Einstein comparison actually came, rather grandiosely, from Joy himself, in an interview with the Post’s Joel Garreau. Except, of course, that Einstein did not advise Roosevelt to renounce development of the bomb, or to end research in atomic physics. In 1939, even pacifist-leaning physicists couldn’t pretend that you could shut down science and count on evildoers to leave it, and you, alone. Facing the Nazis, Joy blinks. Facing Stalin, he closes his eyes altogether.
"We should have learned a lesson from the making of the first atomic bomb and the resulting arms race," writes Joy. "We didn’t do well then, and the parallels to our current situation are troubling."
Joy does not spell out exactly what lesson we should have learned, but by analogy it would be that the atomic bomb and the science underlying it should not have been pursued, because they were simply too dangerous. Physicists should have banded together in the 1930s to shut down this research. It sounds completely nuts, of course, to say that the 20th century’s scientific giants should have abandoned their study of the fundamentals of matter and energy. So, Joy doesn’t say that.