Zombie Statistics
The terrifying power of useful bad data.
"You're about to be untricked," boasted the opening line of a groundbreaking 1981 Reason investigation about high-profile chemical leaks in upstate New York. In the early '80s, Love Canal had already become synonymous with corporate willingness to destroy the environment and human health in the name of profit. But careful reporting revealed the anti-corporate narrative was wrong; the primary malefactor wasn't the greedy businessmen at Hooker Chemical but the Niagara Falls Board of Education, which developed a plot of land despite many warnings from Hooker about the presence of dangerous chemicals. Unfortunately, Reason's story did little to change the anti-market tenor of the environmental reforms that followed.
That's because when a narrative is powerful and useful to highly motivated activists, it can be fiendishly difficult to roll it back. Zombie statistics, in particular, are tough to defeat. These undead tidbits can sustain incredible blows and yet continue to crawl forward, like the plodding, inexorable zombies in George Romero's Night of the Living Dead—a film that debuted in 1968, the same year as Reason. These raggedy facts terrorize the debates over important issues for years after they have been definitively debunked.
At a time when #MeToo and Title IX are dominating the headlines, for instance, it can seem like sexual assault is everywhere. But one of the central statistics responsible for that perception rests on an astonishingly weak foundation. You've probably heard this shocking figure: One in five women has been sexually assaulted while in college.
One of the sources of support for that number is a 2002 study by David Lisak, who concluded that what had previously been referred to as "date rape" was actually the result of repeated infractions by serial campus predators. Lisak urged administrators to view every accusation "as an opportunity to identify a serial rapist," a way of thinking that in turn validates harsh treatment for accused students and justifies funding a massive bureaucracy for adjudication. The Obama White House cited Lisak in memoranda, anti-rape activists promoted his work in movies and books, and university administrators invited him to give lectures and sit on panels.
But as Davidson College administrator Linda M. LeFauve explained in our pages three years ago, Lisak's study was based on survey data cobbled together from his students' dissertations and masters' theses. The central data set drew from interviews with just 76 nontraditional, nonresidential students whose offenses "may or may not have happened on or near a college campus, may or may not have been perpetrated on other students, and may have happened at any time in the survey respondents' adult lives." Despite all these problems, the figure is still widely used and widely believed.
The more horrific and serious-seeming the problem, the less likely anyone is to challenge the data that support calls for action. And no problem seems more dire than human slavery.
Associate Editor Elizabeth Nolan Brown has been tracking the ever-evolving sketchy data that activists and reporters like to wield when it comes to trafficking. As she explained in her 2015 story "The War on Sex Trafficking Is the New War on Drugs," 18 years ago the State Department was claiming that 50,000 people were trafficked into the U.S. each year for forced sex or labor. Over the next decade, the estimate fell as low as 14,500—a 71 percent decrease, utterly unaccounted for in any official documents. The Department of Homeland Security, meanwhile, says that, globally, some 600,000–800,000 people are trafficked across international borders each year. But the Government Accountability Office in 2006 described this figure as "questionable" due to, among other things, the fact that it "was developed by one person who did not document all his work."
What's the only thing more awful than slavery? Child sex slavery. But here again, the most widely cited figures do not add up. Rep. Joyce Beatty (D–Ohio) declared in May 2015 that "in the U.S., some 300,000 children are at risk each year for commercial sexual exploitation." A game of legislative and journalistic telephone followed. The New York Times named the Department of Justice as the source of Beatty's number, while Fox News escalated the estimate to 400,000 and said it came from the Department of Health and Human Services. Both figures, it turns out, are derived from decadesold data analyzed in a paper whose primary researcher no longer endorses his own conclusions.
Everywhere and always in the trafficking debate, the numbers contain a jumble of morally different cases that often conflate everything from the utterly horrific imprisonment and rape of a child against her will, to someone who enters the country illegally after being promised a job as a domestic worker and ends up plucking feathers in a chicken processing plant, to adult sex workers willingly offering their services to adult buyers. At the end of October this year, a magazine ad placed by an airline nonprofit claimed that "every day, over 68,000 victims are trafficked right in front of our eyes, often on commercial flights," a figure basically arrived at by totaling up apples and oranges and dividing by 365.
Campus rape and sex trafficking statistics are like Romero's slow zombies. Their power comes from the fact that they proliferate constantly and never give up their quest for your brain. But sometimes zombie statistics are more like the monsters of more recent flicks such as 28 Days Later and Resident Evil, jumping out at you when you least expect it and moving with surprising speed.
Such is the case with the sudden moral panic over plastic straw use. A fresher variant of the crusade against single-use plastic bags, straws are said to be clogging our waterways and the nostrils of sea turtles everywhere.
The movement from environmentalist meme to active legislation—British Prime Minister Theresa May announced plans for a nationwide ban in April—has been as astonishingly swift as the data behind it are astonishingly weak. In January, Assistant Editor Christian Britschgi was the first to track down the source of the most widespread statistic, that Americans use 500 million plastic straws per day. Turns out that figure came from a nonscientific phone survey conducted by a 9-year-old boy. (Yes, really.)
Britschgi's report was one of Reason's most-read stories for the year, yet the statistic shows no sign of vanishing. In fact, even some publications that noted Britschgi's reporting, such as The Washington Post, lapsed into old habits shortly thereafter. Writers at the Post were back to using the 500 million figure just days later in another part of the paper.
The contagion was powered by celebrities on social media—as a class not known for their fact-checking prowess—from pop stars Demi Lovato and Calvin Harris to five-time Super Bowl champion Tom Brady. Still, municipalities such as Seattle and Santa Barbara have pulled back on laws that would have permitted jail sentences for straw offenders, perhaps due to a rare moment of recognition that bad research should not be the basis for actual deprivations of liberty.
Sometimes good data do creep into the narrative and conventional wisdom does change. Steven Pinker's 2011 book The Better Angels of Our Nature, which tells the story of a long decline in interpersonal violence over the course of human history, was initially greeted with an enormous scoff. But the arguments, buttressed by a tremendous amount of empirical data, have seeped if not into the mainstream then at least into respectable conversation.
Local governments eager for federal grant dollars will do everything in their power to keep the scary numbers about rampant human trafficking alive and active. Environmental activists choose not to look too closely at statistics about plastic use and waste before demanding that you stop drinking your soda with a straw. And advocates for victims of sexual violence are too quick to lean on flawed data that point to a widespread problem when they lobby for more resources.
We are all guilty of this to some extent. It's easy to believe studies that confirm our priors. But Reason has a long history of questioning factoids that seem too good to be true. On global warming, for instance, this magazine was an early voice in the free market coalition to say that the facts were inconvenient, the methodology was tricky to parse, and the resulting policy prescriptions were faulty—but that didn't make the reality of manmade global warming less true.
But the same pattern of employing bad facts for political ends repeats itself, in debates about the wage gap, about the effect of gun ownership levels on gun crime, about child kidnapping, about opioid overdoses, and more.
In so many of these cases, a commonsense fact check should immediately cast suspicion. Are college campuses generally less safe than their surrounding neighborhoods? Does every man, woman, and child in America use 1.5 straws every single day? If child sex trafficking were as common as the more outrageous statistics suggest, Brown calculated, it would mean "nearly 8 million Americans have a robust and ongoing child rape habit, in addition to the alleged millions who pay for sex with adults"—a claim that beggars belief.
The earliest zombies, from Haitian folklore, are revived and sustained by powerful necromancers who deploy the shambling figures for their own dark purposes. So it is with zombie statistics. But we've been keeping our shovels sharp for 50 years here at Reason, and we plan to stay on the job as long as it takes.
This article originally appeared in print under the headline "Zombie Statistics."
