From the late 19th century through the middle of the 20th, futurists imagined electric lighting, but no electric guitars; supersonic jets, but no hang gliders; laser weapons, but no laser surgery or compact disks; giant computer databases, but no Palm Pilots or video games; nuclear power, but no nuclear medicine; government surveillance cameras, but no baby monitors.
These stunted visions -- produced by social critics and science-fiction writers -- are neither random nor isolated. Optimists and pessimists alike conceived of the future -- our present -- as a uniform society, a flattened, unnuanced world designed by a few smart men. They didn't imagine the quirky products of creativity applied to small-scale, personal problems and passions. They didn't factor in the power of vanity, self-expression, chance, novelty, or fun.
Theirs was a future without surprise.
The infatuation with predictability has been deeply imprinted on modern times. From the Communist regimes to corporate giants, we came up in an age of central design, planning, and control. The leading futurists, the science-fiction writers, long depicted progress as the product of elites. In his book "Paris in the Twentieth Century" -- written in 1863, but not published until 1996 -- science-fiction master Jules Verne wrote of "an age when everything was centralized, thought as well as mechanical power."
The unplanned outcomes that emerge from obsessive tinkering, competitive one-upsmanship, incremental improvement, and unarticulated longings have no place in a rigidly planned world. As it turns out, however, they define the world in which we live -- and they will define our future. For as we're now discovering, the future, in fact, is made of surprise.
Even the science-fiction writers nowadays recognize the inevitability of surprise. They "see themselves more as conceptual gardeners, planting for fruitful growth, rather than engineers designing eternal, gray social machines," writes Gregory Benford, the author of such popular science fiction as "Timescape" and "Cosm." "Their views of that future are often playful, seeking to achieve an almost impressionistic effect, imagining small scattered details ... that imply more than they can say."
Small, scattered details aren't just writing techniques. They're also fuel for social and economic propulsion. Important things happen out of sight, often tapping occluded desires. Cultural critics, on the right and left, still argue over where "the '60s" came from, as if someone designed them. Cigar-and-martini-bars, "The Blair Witch Project" and green-and-purple nail polish from Urban Decay, an upstart cosmetics company in California, all took the world by surprise -- unpredicted and unpredictable. So, less fleetingly, did the Web.
What business analyst in the 1970s looked to rural Arkansas to find the future of retailing? Yet that's where Wal-Mart Stores Inc. emerged. It took Jimmy Carter, a born-again Southern Baptist immersed in Bible Belt culture, to recognize the political potential of evangelical voters -- who were there all along.
Technology pundits searched in Silicon Valley for a challenge to Washington state's Microsoft Corp., but never expected the alternative: the free Linux computer operating system, created by hobbyists dispersed around the world who, for fun and hacker prestige, work incessantly to improve it. Linux keeps getting better because thousands of Linux hackers think it's cool to look for bugs. It's an "open source" system, whose code is available to anyone who wants to see it. Linux welcomes ideas and improvements from people anywhere.
"Incessant search by many minds," wrote the late political scientist Aaron Wildavsky, "produces more [and more valuable] knowledge than the attempt to program the paths to discovery by a single one." Mr. Wildavsky, who taught at the University of California, Berkeley, could have been writing about Linux. But the open systems Mr. Wildavsky had in mind were social: science, democracy, markets. These competitive systems encourage scattered knowledge to emerge. They allow for serendipity and thus for surprise.
Surprise drives progress because innovation depends on the sort of knowledge no one can gather in a central place. The Austrian-born economist Friedrich A. Hayek, who won the Nobel Prize in 1974, applied this insight to market prices. In his 1945 article, "The Use of Knowledge in Society," he argued that markets operate as a "system of telecommunications." Prices relay scattered information about what people want, what producers have to offer, and how relative scarcities are changing.
But prices aren't the only way markets transmit information. Markets also allow people with new ideas to test their hypotheses. While other discount retailers concentrated on urban markets, Sam Walton, the founder of Wal-Mart, built stores in small towns, clustering them around central warehouses. He guessed correctly that he had a good model for improving retailing. But only by trying it out, in competition with other retailers, could he be sure.