Refereeing the Debate over the 2021 National Firearms Survey

Critics of Prof. William English's survey sometimes miss the mark, but also raise valid questions.

George Washington professor William English conducted a "2021 National Firearms Survey," which is available on SSRN.com. The latest version, the "expanded report," was published in 2022. In 2024, five professors, led by Harvard's Deborah Azrael authored a critique of the survey, forthcoming in the SMU Law Review. Her coauthors are Joseph Blocher (Duke, law), Philip J. Cook (Duke, public policy), David Hemenway (Harvard, public health), and Matthew Miller (Northeastern & Harvard, public health). In November 2024, English published a detailed response to the critique, as a working paper for the University of Wyoming's Firearms Research Center, where I am a senior fellow. Then in December 2024, Azrael et al. revised their article to include an addendum (pages 24-26) responding English's November paper. In this Post, I will evaluate the pro/con arguments presented by the various authors. [Bracketed inserts] in summaries of what the authors said are by me.

In short:

• English's estimate of a 32% individual adult gun ownership rates is consistent with other surveys.
• His estimates of defensive gun uses (DGUs) — 1.67 million times a year, with a shot fired in about 300,000 — is credible, if one believes that DGUs can be measured at all.
• His age distribution of DGUs seems counterintuitive, and might be contradicted by further studies.
• His estimates of a total national stock of about 44 million rifles that are inaccurately called "assault weapons" could be too high or too low, because of the phrasing of his question. However, other research, when combined with known data other firearms in this category, leads to a minimum national supply easily over 30 million.
• English's estimate that 48% of gun owners own a magazine with 10 round capacity is undisputed.

Estimates of gun ownership

English: "an overall rate of adult firearm ownership of 31.9%."

Azrael: You undercounted people with incomes over $150,000, who are 20% of the population but only 8% of respondents.

English: The "missing rich" are a well-known problem in all surveys. I weighted the results to account for this. The weighting has very little effect on the overall results. E.g., Unweighted: 82.7% of gun owners own a handgun. Weighted: 83.7% own a handgun. Besides, my survey got 1,493 responses from people with incomes over $150,000, whereas Cook's 1994 survey had only 197 responses from people with incomes over $75,000.

Azrael: Your survey may have included "yes" responses from people, particularly females, who live in a household with a gun, but who do not personally own the gun.

English: The question was specifically to individuals, "Do you own any of the following?"

Azrael: Overall ownership rates are consistent with "the high end" of prior surveys. [E.g., Pew Research 2023 = 32%, Gallup Poll 2020 = 32%, Gallup Poll 2021 = 31%, U. Chicago General Social Survey 2021 = 24.5%.] But your female rate is 5% above prior surveys. Maybe that's because you didn't ask a separate question about household rather than individual ownership, and so some females in houses with male-owned guns incorrectly answered "yes."

English: Research indicates there is a large problem of false denial of gun ownership in surveys. [Allison E. Bond et al., Predicting potential underreporting of firearm ownership in a nationally representative sample, 59 Social Psychiatry and Psychiatric Epidemiology 715 (2024).] The Bond article reported a gun ownership rate of 34.6%, and, estimated that false denials (75% chance of actually owning a gun based on demographics) accounted for 6.3% more. Females accounted for over 90% of the false denials. Unlike other surveys, mine used anonymous responses, so that an individual would not be divulging gun ownership at a known address, and my survey didn't mention university affiliation, since respondents may guess that universities are affiliated with a gun control agenda. [Which Harvard, Northeastern, and Duke certainly are, although Duke's Center for Firearms Law does publish guest pieces from diverse perspectives, as does Wyoming.] Also, I included "attention check questions" to weed out respondents who weren't reading the questions carefully.

Azrael reply: "He claims, without evidence, that his survey will encourage more female gun owners to respond honestly. Perhaps, but we might be more convinced if he had eliminated the more compelling explanation for his results by simply asking the single question [household rather than personal ownership] that virtually everyone else has."

Kopel: English did provide an explanation: most importantly, anonymity in responses so that the respondent would know that her home address was unknown to pollster. Plus non-inclusion of university affiliation, which is, in my view, often and accurately perceived as an indicator of anti-gun agenda on the part of the surveyor. The total individual ownership rate is consistent with other surveys, and a mere +5% in the female rate is plausible as a result of survey design. Indeed, it's less than the variation in overall individual rates between other surveys.

"Assault weapons"

English survey question:

"Some have argued that few gun owners actually want or use guns that are commonly classified as 'assault weapons.' Have you ever owned an AR-15 or similarly styled rifle? You can include any rifles of this style that have been modified or moved to be compliant with local law. [E.g., replacing a telescoping stock with a fixed stock.] Answering this will help us establish how popular these types of firearms are."

As English admits, this was a leading question, which he justifies as useful in getting potentially reluctant respondents to disclose ownership: the wording "provided an intelligible explanation to respondents as to why sensitive questions were being asked and avoided language that could alienate respondents in a context where the overriding methodological concern in the literature is under reporting by firearms owners."

The results were: 30.2% of gun owners said that they had ever (not necessarily at present) owned such a gun. This leads to an estimate of 44 million such guns having been owned.

Azrael: The term "assault weapon" is vague [citing Wikipedia]. "The term usually refers not only to certain semi-automatic rifles, but also pistols and shotguns that are able to accept detachable magazines and possess other features. Good surveys make it very clear what they are asking about."

Kopel: Very true. "Assault weapon" bills have covered almost every type of gun, including paintball guns, air guns, most handguns, most rifles, all semiautomatic rifles, most shotguns, and all slide action guns — indeed, almost everything except machine guns. See Kopel, Defining "Assault Weapons", The Regulatory Review (Univ. of Pennsylvania), Nov. 14, 2018.

But, English didn't merely ask about "assault weapon." The specific question was "an AR-15 or similarly styled rifle." This wording addressed Azrael's concern about shotguns and handguns, so it is surprising that she raises this as a rebuttal.

English: "The list of similarly styled rifles that are commonly classified as 'assault weapons' is extensive and would include, for example: AK-47 designs, FN FALs, M1 carbines, M1A's, HK 91/93/94s, SKSs, Kel Tec rifles, Thompsons, and even Ruger 10/22s in certain configurations.

Azrael: "The problem is that while English may know what he meant, many respondents and those relying on his survey may not."

Kopel: Fair point. But the phrasing may well have led to an underestimate of the total ownership rates. For example, the semiautomatic M1 carbine (adopted by U.S. military 1941) and the semiautomatic Thompson rifle (patented 1920) were invented long before the AR-15 (base patent 1956), and look nothing like an AR-15, except in that they are rifles.

Azrael: The 44 million is too high based on other sources: 1. The National Shooting Sports Foundation [trade association for the firearms industry] estimated 24.4 million AR and AK platform rifles manufactured between 1990 and 2020. 2. Washington Post/Ipsos survey of ownership of "AR-15 style rifles" and rifles "built on a common AR-15 platform." 3. Azrael et al. 2023 survey.

English: All the above use narrower definitions. NSSF was solely AR and AK, did not include imports, and did not include the approximately two million homemade AR rifles. [Number estimated by ATF.] Washington Post was solely AR. Both these data sources exclude many models that I included; for example, after World War II, the U.S. government, through the Civilian Marksmanship Program, sold millions of M1 carbines to the American public.  Azrael 2023 incorrectly created the mutually exclusive categories of "semi-automatic military-style rifles" and "semi-automatic hunting rifles."

Kopel: And the 2023 Azrael survey would seem to incline respondents to classify a firearm they own for hunting as not being a "military-style" rifle, even though so-called "assault weapons" are commonly used for hunting. Overall, estimates in the low 20 millions are certainly undercounts. Whether 44 million is the correct figure is uncertain. Azrael is right that "Good surveys make it very clear what they are asking about," and the criticism can be applied equally to the English question and to her own question inaccurately proclaiming that "military-style" and "hunting" are two mutually exclusive categories. One of the most popular hunting rifles in American history is the bolt-action .30-'06 caliber, which was the American military service rifle for the most of the first four decades of the twentieth century; surplus rifles were widely sold to the public via the federal government's Civilian Marksmanship Program.

Defensive gun use

English: My survey indicated that guns are used defensively around "1.67 million times a year: about 300,000 times a shot is fired, about 852,000 times the gun is only brandished, and about 518,000 times neither happens (e.g., someone said they had a gun and that made an aggressor flee)."

Azrael: You should have asked separate questions about DGUs against humans and against animals. Combining the two in one category makes it hard to compare your findings with prior surveys that asked only about use against humans.

Kopel: Fair point.

Azrael: "English includes a type of defensive gun use that is almost always excluded—did you just say you had a gun. He can include these if he wants, but if he wanted his results to be comparable to the many other surveys, all he needed was one question to separate those out."

Kopel: Incorrect. First, English did ask "one question to separate those out." On page 13 of his study, there is the question "Did you fire your gun, show it, or neither." Fired was 18.1%; showed was 50.9%, and neither was 31.0%.

Moreover, Azrael is wrong to claim that English's approach was not "comparable to the many other surveys." The leading prior survey, Gary Kleck 1995, counted a DGU if "the gun was actually used in some way — at a minimum it had to be used as part of a threat against a person, either by verbally referring to the gun (e.g., 'get away — I've got a gun') or by pointing it at an adversary." Gary Kleck & Marc Gertz, Armed Resistance to Crime: The Prevalence and Nature of Self-Defense with a Gun, 86 Journal of Criminal Law and Criminology 150, 162-63 (1995). Kleck found a range of 2.1 to 2.5 million DGUs annually, and those included verbal references. Kleck found that only 24% of DGUs involved firing the gun. This would leave 76% as brandishing or verbal. English went further than Kleck, by asking a question that distinguished brandishing from verbal. Because crime was higher at the time that Kleck conducted his survey, English's 1.67 million DGU figure is consistent with Kleck.

Azrael's co-authors Cook and Hemenway spent years writing journal articles sparring with Kleck about his survey, so they are quite familiar with it. Thus, it is astonishing that an article of which they are coauthors claims that English is an outlier for including verbal references as DGUs. And the claim that English did not ask a question to "separate out" raises concerns about whether the authors of the critique carefully read the paper they are critiquing.

Azrael: English's DGU results are 20 times higher than those reported in the National Crime Victimization Survey. [The NCVS is conducted annually by the Census Bureau in conjunction with the Department of Justice, and is based on in-person interviews.]

English: The NCVS doesn't even ask about DGUs. It just allows persons who have already identified themselves as crime victims to volunteer a DGU in response to an open-ended question about whether they did anything in response to the crime. Even this question is asked only for some crimes. Contrary to Azrael's article, the NCVS does not offer respondents an "option" to indicate a a DGU. [E.g., something such as, "Did you do any of the following: 1. Run away, 2. Use a gun, 3. Use a knife, 4. Try to reason with the attacker."] A RAND Corporation analysis specifically called out these problems with the NCVS as a means of estimating DGUs. "The Challenges of Defining and Measuring Defense Gun Use," Gun Policy in America, Mar. 2, 2018.

Additionally, the NCVS surveyors are specifically identified as working for the U.S. Department of Justice. Respondents may be chary of disclosing DGUs to the federal government.

Kopel: The Azrael NCVS point is preposterous, and at least some of her coauthors can be presumed to be aware of its severe limitations. Notably, those limitations are not mentioned in their article.

The stronger point would have been to adopt the argument in coauthor Cook's 1998 article. There, he conducted a survey that remedied various alleged methodological flaws in the Kleck survey. The data seemed to indicate there were over a million DGUs annually. However, Cook and his coauthor stated, "We find that estimates from this new survey are apparently subject to a large positive bias [i.e., false reporting of a DGU], which calls into question the accuracy of DGU estimates based on data from general-population surveys." Philip J. Cook & Jens Ludwig, Defensive Gun Uses: New Evidence from a National Survey, 14 Journal of Quantitative Criminology 111 (1998).

Thus, one can argue that general population surveys — such as those conducted by English, Kleck, and Cook — are incapable of accurately estimating DGU, and one can also admit that the NCVS, which is also a general population survey that only asks follow-ups to persons who first identify as having been a crime victim, has unique flaws that prevent it from providing a valid estimate of DGUs. Hence, total DGUs are unknowable. This would have been a stronger critique than the one presented by Azrael et al.

Azrael: You asked about lifetime DGUs rather than about DGUs in a specific time period, such as within the last year, or within the last five years.

English: I asked lifetime because Azrael's coauthor Hemenway specifically criticized time-bounded DGU questions as producing artificially high numbers because of the problem of telescoping. That is, if the DGU occurred 13 months ago, it may be such a vivid memory that the respondent recalls it as having happened within the past year. David Hemenway, The Myth of Millions of Annual Self-defense Gun Uses: A Case Study of Survey Overestimates of Rare Events, 10 Chance 6 (no. 3, 1997).

Azrael: "18–20-year-olds in his sample reported more than half (54%) of the total number of lifetime DGUs reported by all age groups combined."

English: Not so. Here are the tables by age: 9,534 total DGUs, with 590 by ages 18-20. English, A Response to Critics, at 18.

Azrael: You're right. We correct the latest version of our article and admit the error in the prior version. (Dec. 5, 2024, version at 19, n.75).

But, your DGUs per respondent are "not credible" because young people have so much higher rates than older people. [DGUs per respondent are .88 for ages 18-20, .90 for ages 21-25, and decline thereafter. Ages 51-55, are .47, and ages 80+ are .15.] "In particular, the 51-55 year olds report only half as many lifetime defensive gun uses as the 21-25 year olds, despite the fact that the older group were young during the most violent period in US history (circa 1990) and have had 30 additional years to add to their experience since then…. Again, if he wanted his to be comparable to all the other surveys, all he needed was one question to ask about whether the most recent defensive gun use was in the past 5 years."

English. "[I]t appears that 80-year-olds passed through their 'most active DGU years' during a time when crime was extremely low, and therefore they accumulated fewer DGU's. Thus, the average of the old and the young taken together approximates the average of the middle aged."

Kopel: It's odd for a paper coauthored by Hemenway to criticize an author for not using the exact technique (asking only about DGUs within the last five years) that Hemenway said should not be used. Damned if you do (Kleck, time-limited), and damned if you don't (English, lifetime).

However, I agree with Azrael that the age distribution of DGUs doesn't make sense. A 75-year-old in his 2021 survey would have been born in 1946, and lived through three major crime waves: The first began in the mid-1960s, began declining after 19800, and surged again starting in the latter 1980s. Another crime decline began in the mid-1990s, and continued through the mid-2010s, by which time crime had fallen to the low levels of the early 1960s. Thereafter, crime sharply increased.

Moreover, according to English, about 25.2% of DGUs occurred inside the home, and 53.9% occurred outside on the home but on the victim's property. English 2022, at 14. Thus, even though today's 70-year-olds would have, in many states, been prevented from carrying a defensive handgun in public for much of their lives, and thus been unable to defend themselves against criminal attackers in a parking lot, those 70-year-olds still would have been able to possess a firearm for defense of their homes, which per English accounts for 79% of DGUs.

Part of the explanation for the higher rates of DGUs reported by young adults in the English survey can be attributed to the fact that violent crime victimization and perpetration is higher for people in their teens and twenties than for older groups. One reason is that they are out and about more, and another reason is their social interactions are more varied. A third reason is younger people are less risk-averse, which makes some of them more likely to perpetrate crimes, and more of them more likely to put themselves in sketchy situations, such as walking around dark city streets very late at night. Conversely, if you're married, in your forties, and living at home with three small children, you're probably don't go out as much at night as you used to, and the people with whom you interact are probably mostly close to your age, which makes them less likely than younger people to perpetrate interpersonal crimes.

Additionally, the younger persons are, the more likely they are to have spent all their adult lives living in jurisdictions where the carrying of defensive arms outside the home is lawful.

As for the upper part of the age cohorts, they entered adulthood at a time when American gun ownership for sport was relatively more common compared to defensive ownership than today. The gun-owner who in 1967 kept an unloaded hunting rifle in a safe in the basement simply might not have been able to access that rifle when burglars broke into his home to take the jewelry and television.

The above analysis is only a partial explanation of what might account for the age skew reported in the English survey. It will be interesting see whether future surveys of DGUs have, like English did, large enough sample size to report results by age, and whether those surveys violate Hemenway's First Law (don't ask time-bound DGU questions) in order to comply with Hemenway's Second Law (don't ask lifetime DGU questions).

Magazines

English: 48% of gun owners have owned a magazine that holds 11 or more rounds. Of them, 62.4% says that home defense is a reason for a choice of a magazine that size.

Azrael: That doesn't prove that "large" magazines are useful for defense. The free-form section of the responses, in which respondents could describe their DGUs, did not include information about how many shots were fired.

English: True, but "Plenty of cases do describe multiple, violent criminal assailants… I leave it for the judicious reader to consider whether being able to fire more than 10 rounds without reloading would or would not be useful in defending against multiple violent assailants.

Kopel: If Azrael et al. sit down next you at a coffee shop and start giving you advice about to weight subgroups in a survey, listen to every word and take copious notes. If they start  advising you how to use your firearm defensively, run away as if your life depended on it.

Next time you see a police officer or sheriff's deputy, take a look at what handgun they are carrying. It will very likely be a pistol with a magazine capacity over 10 rounds. They're not carrying that handgun to shoot woodchucks. They're bearing the gun for the sole purpose of defending innocent lives, including their own. As they well know, most defensive shots, even when fired by law enforcement miss. Further, as they also know, one or two hits usually does not immediately stop an assailant. Additionally, a standard capacity magazine (such as the 11-20 round magazines sold with many pistols) has deterrent value.

Other issues

Kopel: This post discussed the leading issues in the Azrael v. English dispute, but not all of them. I urge readers to read the English survey, the Azrael et al. rebuttal, the English reply and make up their own minds. I have omitted discussion of separate criticism of English's estimate of "assault weapon" prevalence by professor Louis Klarevas, of Columbia Teacher's College, because Klarevas evidently misread the English paper. A few respondents had claimed to own an implausible number of "AR-15 style" rifles, such as 1,000. English dropped these responses from his estimates, but Klarevas believed that these numbers were included. English 2024, at 20-23.