Mother Jones Claims the Rate of Mass Shootings Has Tripled Since 2011. Is That True?

A new bottle for some old data

Two years ago, Mother Jones published a "Guide to Mass Shootings in America," which claimed to show that such slayings happen more often now than in the past. The feature, whose data have been updated periodically since it first went up, has been both widely cited and harshly disputed. This week the outlet has published a new analysis claiming that mass shootings occur far more frequently than they did several decades ago and that the rate has tripled since 2011. Whether you buy that depends on whether you buy into the previous piece, because the new article has essentially the exact same flaws.

The best critique I've seen of the original Mother Jones article was written by Michael Siegel. You should read the whole thing, but this is the key passage:

It is a truism of science that the more narrowly you define your sample and the more you shrink the number of data points, the less reliable your conclusions will be. If you were to analyze all gun shootings and violence over the last thirty years, you'd have hundreds of thousands of data points to base your conclusions on. You could, as I like to say, achieve Victory Through Sheer Data Volume. But when you start parsing the data down further and further, you become more prone to random variation and even bias.

Even if we take Mother Jones' data at face value, we can see we're dealing with less than 120 victims every year and frequently less than 20. That's an awfully small number to be drawing conclusions from. To illustrate why, take the Virginia Tech killings. 56 people were killed or wounded. That is more than all but five entire years in their database. Something like that is simply going to swamp the statistics.

But we shouldn't even take Mother Jones' data at face value because it is highly suspect. First, it seems to be based on media coverage, which is not exactly an objective source and almost certainly leaves shootings out….Everywhere, they make arbitrary cuts to exclude murders that may not fit their conclusions. They limit the sample to lone shooters, but make exceptions for Columbine and Westside. They exclude gang activity and other crimes but include the Fort Hood Shootings, which were an act of terrorism….They arbitrarily throw in a few spree killings.

This is simply not a representative sample. It's cherry-picked to fit a definition, but leaves huge gaping biases all over the place. Mother Jones doesn't even acknowledge this.

All this would be fine if you wanted to create an illustrative or representative sample. This is even fine if you want to draw some broad and overwhelming conclusions such as that most spree killers get their guns legally. But the low numbers and the biases blow up in your face when you try to do a more rigorous analysis….They've narrowed the sample so far down that they are essentially looking at noise.

The new analysis looks at the intervals between each incident rather than the annual numbers of crimes and victims. But aside from the fact that the list has been updated through 2014, this is the same data as the original article, with all the problems that Siegel and others pointed out before.

The Mother Jones team does make one reasonable point, in the course of arguing against those of us who aren't convinced mass shootings have been getting more common:

So why do we keep hearing in the media that mass shootings have not increased?

This view stems from the work of Northeastern University criminologist James Alan Fox, who has long maintained that mass shootings are a stable phenomenon. ("The growing menace lies more in our fears than in the facts," he has said.) But Fox's oft-cited claim is based on a misguided approach to studying the problem: The data he uses includes all homicides in which four or more people were murdered with a gun. His analysis, which counts the number of events per year, lumps together mass shootings in public places with a far more numerous set of mass murders that are contextually distinct—a majority of which stem from domestic violence and occur in private homes.

It is true that Fox's data cover a lot of incidents that don't fit the standard conception of a mass shooting, and that it would be useful to have a more narrowly defined count. The flipside of this is that Fox's figures are based on a relatively solid source—police reports, which are regularly collected and tabulated—and thus avoid the problems with relying on press accounts for your list of incidents.

The best alternative measurement that I'm aware of comes from Grant Duwe, a criminologist at the Minnesota Department of Corrections. His definition of mass public shootings does not make the various one-time exceptions and other jerry-riggings that Siegel criticizes in the Mother Jones list; he simply keeps track of mass shootings that took place in public and were not a byproduct of some other crime, such as a robbery. And rather than beginning with a search of news accounts, with all the gaps and distortions that entails, he starts with the FBI's Supplementary Homicide Reports to find out when and where mass killings happened, then looks for news reports to fill in the details. According to Duwe, the annual number of mass public shootings declined from 1999 to 2011, spiked in 2012, then regressed to the mean.

Finally, a note on why this matters. Violent crime rates have been moving downward for decades now, and mass shootings—by any definition—are a very rare phenomenon. I've heard arguments from one direction that say there's no point in putting such a small risk under a microscope when the most pressing threats to people's lives lie elsewhere. I've heard arguments from another direction that say even one crime this horrible is too many, and that the effect of noting how infrequently it happens is just to discourage people from trying to prevent it.

To the first set of arguments, I say that when the press and politicians present a problem like this as a rising crisis, it's worthwhile to see whether it is indeed rising. To the second set of arguments, I say that absolutely nothing I've said here means we shouldn't try to prevent future mass murders. Plane crashes are extremely rare, but airlines still look for ways to make them even less likely. If a measure genuinely makes people safer without creating an intolerable trade-off, I'm for it.

Such measures are most likely to be incremental changes adopted at particular places (such as schools) and then imitated elsewhere, not big anti-crime bills rushed into law by national politicians eager to be seen Doing Something. But there may well be ways federal or state officials can make that experimentation and imitation easier. Good ideas are good—and bad ideas are bad—whether or not mass shootings are getting more common.

Past Reason coverage of the issue:

• "Are Mass Shootings Becoming More Common in the United States?" (December 2012)

• "Making Sense of Mass-Shooting Statistics" (January 2013)

• "Life During Wartime" (May 2013)

• "Why Can't Anyone Agree How Many Mass Shootings There Have Been In 2013?" (September 2013)

• "Why Mass Shootings Haven't Ushered In a New Age of Gun Control" (February 2014)

• "Are School Homicides 'Becoming the Norm'?" (June 2014)

• "The FBI Says 'Active Shooter Incidents' Are On the Rise. What Does That Mean?" (September 2014)