Why is success so hard to predict?

"Star Wars" has become the most successful movie saga of all time. Total franchise revenue exceeds $30 billion. But before its release, no one predicted anything like that. In fact, almost everyone thought that it would be a big fat flop. (A bit like the Harry Potter novels and Donald Trump's candidacy.) What happened? How could people have been so wrong?

A few years ago, social scientists Matthew Salganik, Duncan Watts and Peter Dodds sought to answer the question of why cultural success is so unpredictable. They created an artificial music market on an actual website, which they called the Music Lab. The site offered people an opportunity to hear 48 unknown songs by unknown bands.

The experimenters randomly sorted half of about 14,000 site visitors into an "independent judgment" group, in which they were invited to listen to brief excerpts, to rate songs and to decide whether to download them. For 7,000 or so visitors, Salganik and his co-authors could get a clear sense of what people really liked best. The other 7,000 visitors were sorted into a "social influence" group, which was exactly the same except in just one respect: They could see how many times each song had been downloaded by other participants.

Here's the ingenious part of the experiment: People in the social influence group were also randomly assigned to one of eight subgroups, in which they could see only the number of downloads in their own subgroup. In those different subgroups, it is inevitable that as a result of random factors, different songs would attract different initial numbers of downloads.

The research question was this: Would the initial numbers matter to where songs would ultimately end up on the researchers' hit parade? You might expect that quality would always prevail—that the popularity of the songs, as measured by their download rankings, would be roughly the same in the independent group and in all eight of the social influence groups.

But that isn't what happened—not at all. In short, everything turned on initial popularity. Almost any song could do really well or really poorly, depending on whether or not the first visitors liked it. Importantly, there was one qualification: the songs that did the very best in the independent group rarely did very badly, and the songs that did the very worst in the independent group rarely did spectacularly well. But otherwise, almost anything could happen.

Here's a cautious, modest reading of these findings. Some products really are destined for success, and others really are destined for failure. If a song is truly sensational, it will be a hit. Mozart, Shakespeare, Dickens and Taylor Swift were bound for success (and the same might be true for "Star Wars"). If a song sounds horrible, it's going to flop. But within a wide range, songs can do very well or very poorly, and within that range, you just can't predict.

This is a plausible reading of the Music Lab experiment, but I think that it is far too cautious. Sure, terrible songs, movies and books are unlikely to succeed. But maybe the best ones are not destined for success. Almost nothing is. After all, the Music Lab experiment itself was tightly controlled. It had had just 48 songs. In real markets there are countless more. And in those real-world markets, media attention, critical acclaim, marketing and product placement play a big role.

If you aren't persuaded, let's turn away from experiments to the real world. If any literary figures are great, the short list must include William Wordsworth, John Keats, Jane Austen and William Blake. Equally surely, it does not include George Crabbe, Robert Southey, Barry Cornwall, Leigh Hunt and Mary Brunton. But H. J. Jackson's important study of literary reputation, "Those Who Write For Immortality" strongly suggests that even for the greatest of the great, it's a lot like the Music Lab: Accident, contingency and luck play a massive role.

In terms of perceived quality, Wordsworth, Crabbe and Southey were grouped together during their lifetimes. The same is true of Keats, Cornwall and Hunt, and also of Austen and Brunton. Jackson's conclusion, which she makes quite plausible, is that "[a]s far as reputation is concerned, the differences between them are largely personal and accidental." Indeed, their contemporaries put Cornwall far above Keats—and the now-obscure Hunt ranked above them both. And if we are interested in professional opinions, we will find that Wordsworth, Samuel Taylor Coleridge and Lord Byron all ranked Cornwall highest of the three.

In their time, Brunton and Austen were about equally well regarded, but the former, of course, faded into obscurity. Jackson urges, and demonstrates, that what "happened to Brunton—the gradual fading and extinction of her name—could easily have happened to Austen." Jackson's conclusion is that "long-term survival has depended more on external circumstances and accidental advantages than on inherent literary worth."

Even for those who love them, "Star Wars" isn't quite Wordsworth or Austen, and we don't know yet know about its long-term survival. But $30 billion is a lot of money. Has its triumph been a product of external circumstances and accidental advantages? In the end, I don't think so; its inventiveness and exuberance ensured success. But on that question, it would be a mistake to have much confidence. For our great cultural icons—and for products, politicians and ideas as well—the Music Lab is an important reminder: It could easily have been otherwise.