A Second Look at a Controversial Study About Defensive Gun Use

Criminologist Gary Kleck revises his paper on the incidence of the use of firearms for self-protection.

In April, criminologist Gary Kleck reported that he had uncovered evidence supporting his contention that Americans use firearms in self defense over 2 million a times a year. The survey he discovered had not been previously analyzed, but he reported that it matched what he found in the 1993 survey he conducted with Marc Gertz and published in 1995, known as the National Self-Defense Survey (NSDS).

His new report was based on surveys conducted by the Centers for Disease Control (CDC) in its Behavioral Risk Factor Surveillance System (BRFSS) survey in the years 1996-98. This finding was touted by many outlets—including Reason—as evidence in support of the utility of private gun ownership.

Shortly after that study was released, however, Robert VerBruggen of National Review (who has been of inestimable help in thinking through these issues) tweeted that he noticed, by studying the raw survey data himself, that Kleck had mistaken what were in fact surveys limited to small numbers of states per year for a national survey, analogous to Kleck/Gertz's own national surveys.

in direct response to queries from Reason, who first directly notified Kleck of his error, he worked through and has since issued a revised version of the paper, published as was the original as a working paper on the Social Science Research Network. In the new version, Kleck re-analyzes the BRFSS survey data accurately as limited to a small number of states, and ultimately concludes, when their surveys are analyzed in conjunction with his NSDS, that their surveys indicate likely over 1 million defensive uses of guns (DGUs) a year nationally, compared to the over 2 million of his own NSDS.

Here's how Kleck got to that new conclusion. The BRFSS, as Kleck describes it in his paper, "are high-quality telephone surveys of very large probability samples of U.S. adults…even just the subset of four to seven state surveys that asked about DGU in 1996-1998 interviewed 3,197-4,500 adults, depending on the year. This is more people than were asked about this topic in any other surveys, other than the National Self-Defense Survey conducted in 1993 by Kleck and Gertz (1995), who asked DGU questions of 4,977 people." The BRFSS asked about defensive uses of guns in seven states in 1996, seven in 1997, and four in 1998.

Kleck judged the "wording of the DGU question in the BRFSS surveys" as "also excellent, avoiding many problems with the wording that afflicted the DGU questions used in other surveys."

The BRFSS results were designed to exclude "uses by military, police and others with firearm-related jobs" and "uses against animals." The survey was designed to garner "yes" answers as long as a gun was used in presumed self-defense in any location (not just the home), whether or not the gun was actually fired (as, per Kleck's survey, around 3/4 of the time one needn't fire the gun to have found it useful in deterring an intruder or attacker).

Since Kleck's survey did not include Alaska and Hawaii and the BRFSS did (in 1996 and 1997 respectively), he kept them out of the comparison. The states for which a meaningful comparison could be made between his NSDS and this CDC survey, then, were, in 1996, Kentucky, Louisiana (also surveyed in 1998), Maryland, New Hampshire (also surveyed in 1997), New York, and West Virginia; in 1997, Colorado, Missouri, New Jersey (also surveyed in 1998), North Dakota, and Ohio; and in 1998, Montana and Pennsylvania.

Kleck notes that it's simply impossible to extrapolate meaningfully from the small set of states surveyed over the course of those three years to a solid national DGU figure from the BRFSS itself: "We cannot directly apply these estimates to the U.S. because the sets of states do not constitute a probability sample of the U.S. The prevalence of DGU could be far higher in some states than in the nation as a whole if the states have higher-than-average rates of gun ownership and/or crime, or could be far lower if the set of states had lower gun ownership or crime rates."

But he does think by comparing the national results from his NSDS to the results in the BRFSS-surveyed set of states in his NSDS you can make a tentative extrapolation (after adjusting for the fact that the BRFSS only asked the DGU question to households that had already said they had a gun, while his surveys "found that 21% of persons who reported a DGU had denied having a gun in their household at the time of the interview.")

In the group of states (minus Alaska and Hawaii) that BRFSS surveyed over those three years, the BRFSS found raw numbers of 55 (1996), 29 (1997), and 33 (1998) DGUs.

After a series of adjustments and weightings described at length in the paper, Kleck concludes the BRFSS survey indicates that the percentage of adults in gun-owning households who experienced a DGU in the states they surveyed were 1.33 percent for 1996, 0.89 percent for 1997, and 1.04 percent for 1998.

Again, while a straight national extrapolation for the BRFSS data alone can't be meaningfully done, Kleck tries, presuming that the ratio of the national DGU rate over the rate in the specific group of states that BRFSS happened to survey found in his NSDS should hold for the BRFSS as well, to make a national DGU rate guess from the BRFSS data. He ends up calculating national percentage rates for adults in gun-owning households nationally of 0.59 percent based on the 1996 states, 0.81 percent based on the 1997 states, and 1.82 percent based on the 1998 states.

After adjustments to get a guess for total adults, not just adults in gun-owning households, the range of total DGUs Kleck estimates for the nation with the above methods from the CDC's state-level surveys range from a low of 620,648 for 1996 to 1.9 million in 1998, for an average over the years of 1.1 million.

In Kleck and Gertz's NSDS, 10, 8, and 4 "weighted past year DGU cases" were found in the states BRFSS surveyed in the years 1996, '97, and '98 respectively. Given that very small number of actual pre-extrapolation DGUs Kleck found in the states that both he and the BRFSS covered in 1998, just four, making extrapolative adjustments based on them might seem to overstate his case.

As the adjustments work, for example, as spelled out for me initially by VerBruggen, had Kleck/Gertz found just two more DGUs in their surveys over the four 1998 states, the adjustments downward for the ratio of total U.S. DGUs over that group of states would be from 1.7 to 1.1, meaning that the national extrapolation for the BRFSS based on the NSDS for that year would be a whole number around 600,000 DGUs lower, and for that three-year average around 200,000 lower. That seems a lot of weight to place on such a tiny initial count. (There is also the wrinkle, as drawn out by economist Alex Tabarrok when discussing Kleck's first version of this paper, that with surveys regarding rare events, even a small percentage of liars, if the lies are distributed without any particular bias one way or the other, can very much overstate the phenomenon.)

A CDC representative, when asked about why no study using or publicizing the raw data on this DGU survey was ever issued, a matter Kleck speculates on quite a bit, wrote merely that "Data from the optional module data [asking the questions for those years' BRFSS were optional to the states, which is why only a few did] were made available to the public to analyze via the BRFSS public use dataset online" which is where Kleck eventually found them, though CDC never otherwise drew any conclusions from them in any publication nor drew anyone's attention to them.