You Know Less Than You Think About Guns

The misleading uses, flagrant abuses, and shoddy statistics of social science about gun violence

Paul Chesley/Getty Images

"There is a gun for roughly every man, woman, and child in America," President Barack Obama proclaimed after the October mass shooting that killed 10 at Umpqua Community College in Oregon. "So how can you, with a straight face, make the argument that more guns will make us safer? We know that states with the most gun laws tend to have the fewest gun deaths. So the notion that gun laws don't work—or just will make it harder for law-abiding citizens and criminals will still get their guns—is not borne out by the evidence."

In this single brief statement, Obama tidily listed the major questions bedeviling social science research about guns—while also embodying the biggest problem with the way we process and apply that research. The president's ironclad confidence in the conclusiveness of the science, and therefore the desirability of "common-sense gun safety laws," is echoed widely with every new mass shooting, from academia to the popular press to that guy you knew from high school on Facebook.

In April 2015, the Harvard gun-violence researcher David Hemenway took to the pages of the Los Angeles Times to declare in a headline: "There's scientific consensus on guns—and the NRA won't like it." Hemenway insisted that researchers have definitively established "that a gun in the home makes it a more dangerous place to be…that guns are not used in self-defense far more often than they are used in crime…and that the change to more permissive gun carrying laws has not reduced crime rates." He concludes: "There is consensus that strong gun laws reduce homicide."

But the science is a lot less certain than that. What we really know about the costs and benefits of private gun ownership and the efficacy of gun laws is far more fragile than what Hemenway and the president would have us believe.

More guns do not necessarily mean more homicides. More gun laws do not necessarily mean less gun crime. Finding good science is hard enough; finding good social science on a topic so fraught with politics is nigh impossible. The facts then become even more muddled as the conclusions of those less-than-ironclad academic studies cycle through the press and social media in a massive game of telephone. Despite the confident assertions of the gun controllers and decades of research, we still know astonishingly little about how guns actually function in society and almost nothing at all about whether gun control policies actually work as promised.

Do More Guns Mean More Homicides?

"More Americans have died from guns in the United States since 1968 than on battlefields of all the wars in American history," New York Times columnist Nicholas Kristof wrote on August 26, 2015, just after the grisly on-air murder of two television journalists in Virginia. It's a startling fact, and true.

But do the number of guns in circulation correlate with the number of gun deaths? Start by looking at the category of gun death that propels all gun policy discussion: homicides. (Gun suicides, discussed further below, are a separate matter whose frequent conflation with gun crime introduces much confusion into the debate.)

In 1994 Americans owned around 192 million guns, according to the U.S. Justice Department's National Institute of Justice. Today, that figure is somewhere between 245 and 328 million, though as Philip J. Cook and Kristin A. Goss in their thorough 2014 book The Gun Debate: What Everyone Needs to Know (Oxford University Press) wisely concluded, "the bottom line is that no one knows how many firearms are in private hands in the United States." Still, we have reason to believe gun prevalence likely surpassed the one-gun-per-adult mark early in President Barack Obama's first term, according to a 2012 Congressional Research Service report that relied on sales and import data.

Yet during that same period, per-capita gun murders have been cut almost in half.

One could argue that the relevant number is not the number of guns, but the number of people with access to guns. That figure is also ambiguous. A Gallup poll in 2014 found 42 percent of households claiming to own a gun, which Gallup reports is "similar to the average reported to Gallup over the past decade." But those looking for a smaller number, to downplay the significance of guns in American life, can rely on the door-to-door General Social Survey, which reported in 2014 that only 31 percent of households have guns, down 11 percentage points from 1993's 42 percent. There is no singular theory to explain that discrepancy or to be sure which one is closer to correct—though some doubt, especially as gun ownership continues to be so politically contentious, that people always reliably report the weapons they own to a stranger literally at their door.

The gun murder rate in 1993 was 7.0 per 100,000, according to the Centers for Disease Control and Prevention's (CDC) National Center for Injury Prevention and Control. (Those reports rely on death certificate reporting, and they tend to show higher numbers than the FBI's Uniform Crime Reporting program, though both trend the same.) In 2000 the gun murder rate per 100,000 was 3.8. By 2013, the rate was even lower, at 3.5, though there was a slight upswing in the mid-00s.

This simple point—that America is awash with more guns than ever before, yet we are killing each other with guns at a far lower rate than when we had far fewer guns—undermines the narrative that there is a straightforward, causal relationship between increased gun prevalence and gun homicide. Even if you fall back on the conclusion that it's just a small number of owners stockpiling more and more guns, it's hard to escape noticing that even these hoarders seem to be harming fewer and fewer people with their weapons, casting doubt on the proposition that gun ownership is a political crisis demanding action.

In the face of these trend lines—way more guns, way fewer gun murders—how can politicians such as Obama and Hillary Clinton so successfully capitalize on the panic that follows each high profile shooting? Partly because Americans haven't caught on to the crime drop. A 2013 Pew Research Poll found 56 percent of respondents thought that gun crime had gone up over the past 20 years, and only 12 percent were aware it had declined.

Do Gun Laws Stop Gun Crimes?

The same week Kristof's column came out, National Journal attracted major media attention with a showy piece of research and analysis headlined "The States With The Most Gun Laws See The Fewest Gun-Related Deaths." The subhead lamented: "But there's still little appetite to talk about more restrictions."

Critics quickly noted that the Journal's Libby Isenstein had included suicides among "gun-related deaths" and suicide-irrelevant policies such as stand-your-ground laws among its tally of "gun laws." That meant that high-suicide, low-homicide states such as Wyoming, Alaska, and Idaho were taken to task for their liberal carry-permit policies. Worse, several of the states with what the Brady Campaign to Prevent Gun Violence considers terribly lax gun laws were dropped from Isenstein's data set because their murder rates were too low!

Another of National Journal's mistakes is a common one in gun science: The paper didn't look at gun statistics in the context of overall violent crime, a much more relevant measure to the policy debate. After all, if less gun crime doesn't mean less crime overall—if criminals simply substitute other weapons or means when guns are less available—the benefit of the relevant gun laws is thrown into doubt. When Thomas Firey of the Cato Institute ran regressions of Isenstein's study with slightly different specifications and considering all violent crime, each of her effects either disappeared or reversed.

Another recent well-publicized study trying to assert a positive connection between gun laws and public safety was a 2013 JAMA Internal Medicine article by the Harvard pediatrics professor Eric W. Fleegler and his colleagues, called "Firearm Legislation and Firearm-Related Fatalities in the United States." It offered a mostly static comparison of the toughness of state gun laws (as rated by the gun control lobbyists at the Brady Center) with gun deaths from 2007 to 2010.

"States with strictest firearm laws have lowest rates of gun deaths," a Boston Globe headline then announced. But once again, if you take the simple, obvious step of separating out suicides from murders, the correlations that buttress the supposed causations disappear. As John Hinderaker headlined his reaction at the Power Line blog, "New Study Finds Firearm Laws Do Nothing to Prevent Homicides."

Among other anomalies in Fleegler's research, Hinderaker pointed out that it didn't include Washington, D.C., with its strict gun laws and frequent homicides. If just one weak-gun-law state, Louisiana, were taken out of the equation, "the remaining nine lowest-regulation states have an average gun homicide rate of 2.8 per 100,000, which is 12.5% less than the average of the ten states with the strictest gun control laws," he found.

Public health researcher Garen Wintemute, who advocates stronger gun laws, assessed the spate of gun-law studies during an October interview with Slate and found it wanting: "There have been studies that have essentially toted up the number of laws various states have on the books and examined the association between the number of laws and rates of firearm death," said Wintemute, who is a medical doctor and researcher at the University of California, Davis. "That's really bad science, and it shouldn't inform policymaking."

Wintemute thinks the factor such studies don't adequately consider is the number of people in a state who have guns to begin with, which is generally not known or even well-estimated on levels smaller than national, though researchers have used proxies from subscribers to certain gun-related magazines and percentages of suicides committed with guns to make educated guesses. "Perhaps these laws decrease mortality by decreasing firearm ownership, in which case firearm ownership mediates the association," Wintemute wrote in a 2013 JAMA Internal Medicine paper. "But perhaps, and more plausibly, these laws are more readily enacted in states where the prevalence of firearm ownership is low—there will be less opposition to them—and firearm ownership confounds the association."

What About Suicides?

Removing suicides from "gun deaths" is a basic step for assessing whether a gun regulation is producing its proposed effect, which in most cases is to reduce the number and severity of gun murders. But what do gun suicide rates tell us on their own?

Chiefly, that a gun is a very efficient means of killing yourself. According to the CDC's National Vital Statistics System, 21,175 Americans committed suicide with firearms in 2013, more than twice as many as used the next most popular suicide method, suffocation. There were nearly twice as many gun suicides that year as gun homicides.

Gun owners are more than three times as likely to commit suicide as non-gun owners, according to a 2014 Annals of Internal Medicine meta-analysis by Andrew Anglemyer and his colleagues. They looked at 14 previous observational studies regarding suicide from 1988 to 2005, statistically re-analyzing them all together. They found that the studies (with one exception) indicated that the people who committed suicide (whether with a gun or not) were more likely, usually far more likely, to own guns than the control group of people with similar characteristics who did not kill themselves. This does not, however, allow us to conclude that the gun's presence caused the suicide, since it's always possible that those more likely to be suicidal are more likely to want to own guns.

A 2002 study by Mark Duggan, now an economist at Stanford University, seems to endorse that conclusion, writing that "much of the positive relationship between firearms ownership and suicide is driven by selection—individuals with above average suicidal tendencies are more likely to own a gun and to live in areas with relatively many gun owners."

The U.S. currently ranks 47th in total suicide rates among nations according to World Health Organization (WHO) calculations, and 11th among Organization for Economic Co-operation and Development nations. But our firearm suicide rates are among the highest in the world, likely behind only Uruguay. Nations with far tougher gun laws and far lower known prevalences of gun ownership, such as Japan, India, and Korea, have far higher overall suicide rates. This suggests that the percentage of firearms in America leads us to have more firearm suicides, but not necessarily more suicides overall.

Of the 56 nations for which the WHO felt it had accurate reported method data, hanging remained the most popular means of death, accounting for over 40 percent of suicides in 35 of them. At least one study—"Small Arms Mortality: Access to Firearms and Lethal Violence," by Mark Konty and Brian Schaefer, published in 2012 in the journal Sociological Spectrum—used "nation-level…data from the Small Arms Survey and the World Health Organization's measures of mortality" to "examine whether rates of small arm ownership have a positive effect on rates of homicide and suicide." Their conclusion: "Contrary to the opportunity model, the accessibility of firearms does not produce more homicide or suicide when other known factors are controlled for."

Still, evidence from the Anglemyer meta-analysis suggests that policies like waiting periods, trigger locks, or other "safe storage" requirements might prevent some suicides by inserting at least a little extra time to think things through.

Is Having a Gun in the Home Inherently Deadly?

The idea that keeping a gun in the home puts owners and their families at elevated risk first rose to prominence in a 1993 New England Journal of Medicine article by Arthur Kellermann and his colleagues. "Although firearms are often kept in homes for personal protection," they concluded, "this study shows that the practice is counterproductive."

The study has many flaws. In addition to the predictable failure to establish causality, there's a more glaring irregularity: Slightly less than half of the murders Kellermann studied were actually committed with a gun (substantially less than the national average in 1993 of around 71 percent). And even in those cases he failed to establish that the gun owners were killed with their own guns. If even a small percentage of them weren't, given that more than half of the murders were not committed with guns, the causal relevance of the harmed being gun owners is far less clear. (The study found that even more dangerous risks than having a gun at home included living alone, using drugs, or being a renter.)

A 2013 literature review in the journal Aggression and Violent Behavior, written by the University of Utrecht psychologist Wolfgang Stroebe, starts with Kellermann but rejects the idea that firearm possession is "a primary cause of either suicide or homicide." However, he writes, "since guns are more effective means for [actually killing someone] than poison or other weapons, the rate of firearm possession can be expected to be positively related to overall rates of suicide and homicide." But even then we can't be sure of causality, since guns might be the choice of people with more serious lethal intent, against themselves or others, to begin with.

Stroebe notes that the two major post-Kellermann studies most often used to demonstrate an association between gun ownership and risk of homicide shared one of Kellermann's fatal flaws: They offer no information about whether the gun used to kill the gun owners was their own. And despite Kellermann's finding that living alone was very risky, one of the follow-ups, a 2004 study by Linda Dahlberg and colleagues, found that it was only those with roommates who faced a higher risk of a specifically gun-related homicide.

Are Guns a Public Health Hazard?

Public health—long associated with the prevention of communicable diseases—got into the gun social science game with a vengeance in the 1990s. These scholars commonly viewed weapons as nothing more than vectors for harm; leading lights, such as a professor at Harvard's School of Public Health, could happily declare: "I hate guns and cannot imagine why anybody would want to own one. If I had my way, guns for sport would be registered, and all other guns would be banned." The CDC earlier in 1987 published a study openly recommending confiscating guns in the name of public health.

Public health scholars have continued to research from a place of hostility to firearms. An October 2015 special issue of the journal Preventive Medicine dedicated to guns began with an editorial that praised the role the public health movement played in spreading vaccines and reducing tobacco use, then cut to the quick: "It is the editorial position of this journal that there is one overtly visible and low-hanging fruit left in the tree, one that has surprisingly eluded concerted action from public health: gun violence prevention." Alas, there is an obstacle: the "peculiar proclivity that much of the American population has with firearms."

That proclivity is indeed vast. In addition to those owning guns for reasons of self-defense, there is the massive recreational component. A 2011 U.S. Fish and Wildlife Service survey found that "13.7 million people, 6% of the U.S. population 16 years old and older, went hunting." The National Sporting Goods Association says there were at least 20 million recreational target shooters in the U.S. as of 2014.

Less quantifiable, but still quite real, are the sense of self-fulfillment and identity that guns and gun culture bring to Americans, the same way any other recreation from surfing to sailing to car culture does. Attempts to scientifically demonstrate the "social costs" of guns—for example, a 2006 Journal of Public Economics paper called "The Social Costs of Gun Ownership," by Duke's Philip Cook and Jens Ludwig (then of Georgetown)—typically don't rigorously address these benefits.

While most of the articles in the Preventive Medicine issue were standard anti-gun material, one piece perhaps inadvertently undermined a popular argument for expanding background checks. "Sources of Guns to Dangerous People: What We Learn By Asking Them," by Philip Cook and colleagues, surveyed a set of jailed criminals in Cook County, Illinois. It found that they "obtain most of their guns from their social network of personal connections. Rarely is the proximate source either direct purchase from a gun store, or theft." So the go-to remedy for gun control advocates seeking to limit homicides might not have much impact on actual gun criminals.

How Often Are Guns Used Defensively?

One of the most powerful narratives gun advocates have on their side is the image of a woman pulling a handgun out of her clutch to prevent a rape, or a man cocking a shotgun at a burglar to defend his family.

Many social scientists who research this issue of "defensive gun use" (DGUs) say such scenarios are vanishingly rare, arguing that owning a gun is more likely to lead to harm for the owner than be his or her savior in a pinch.

There are no even halfway thorough documentations of every such event in America. They are not all going to end up reported in the media or to the police. The FBI and the CDC will have no reason to record or learn about the vast majority of times a crime was prevented by the potential victim being armed. So our best estimates come from surveys.

The survey work most famous for establishing a large number of DGUs—as many as 2.5 million a year—was conducted in 1993 by the Florida State University criminologists Gary Kleck and Marc Gertz. Kleck says they found 222 bonafide DGUs directly via a randomized anonymous nationwide telephone survey of around 5,000 people. The defender had to "state a specific crime they thought was being committed" and to have actually made use of the weapon, even if just threateningly or by "verbally referring to the gun." Kleck insists the surveyors were scrupulous about eliminating any responses that seemed sketchy or questionable or didn't hold up under scrutiny.

Extrapolating from their results, Kleck and Gertz concluded that 2.2 to 2.5 million DGUs happened in the U.S. each year. In a 2001 edition of his book Armed, Kleck wrote that "there are now at least nineteen professional surveys, seventeen of them national in scope, that indicate huge numbers of defensive gun uses in the U.S." The one that most closely matched Kleck's methods, though the sample size was only half and the surveyors were not experienced with crime surveys, was 1994's National Survey of the Private Ownership of Firearms. It was sponsored by the U.S. Justice Department and found even more, when explicitly limiting them to ones that met the same criteria as Kleck's study—4.7 million (though the research write-up contains some details that may make you wonder about the accuracy of the reports, including one woman who reported 52 separate DGUs in a year).

The major outlier in the other direction, nearly always relied on for those downplaying the defensive benefits of guns, is the Bureau of Justice Statistics' National Crime Victimization Survey (NCVS), a nationally representative telephone survey, which tends to find less than 70,000 DGUs per year.

In the October 2015 special issue on "gun violence prevention," Preventive Medicine featured the latest and most thorough attempt to treat the NCVS as the gold standard for measuring defensive gun usage. The study, by Harvard's Hemenway and Sara J. Solnick of the University of Vermont, broke down the characteristics of the small number of DGUs recorded by the NCVS from 2007 to 2011. The authors found, among other things, that "Of the 127 incidents in which victims used a gun in self-defense, they were injured after they used a gun in 4.1% of the incidents. Running away and calling the police were associated with a reduced likelihood of injury after taking action; self-defense gun use was not." That sounds not so great, but Hemenway went on to explain that "attacking or threatening the perpetrator with a gun had no significant effect on the likelihood of the victim being injured after taking self-protective action," since slightly more people who tried non-firearm means of defending themselves were injured. Thus, for those who place value on self-defense and resistance over running, the use of a weapon doesn't seem too bad comparatively; Hemenway found that 55.9 percent of victims who took any kind of protective action lost property, but only 38.5 percent of people who used a gun in self-defense did.

Kleck thinks the National Crime Victimization Survey disagrees so much with his own survey because NCVS researchers aren't looking for DGUs, or even asking about them in so many words. The survey merely asks those who said "yes" to having been a crime victim whether they "did or tried to do" something about it. (You might not consider yourself a "victim" of a crime you have successfully prevented.) Kleck surmises that people might be reluctant to admit to possibly criminal action on their own part (especially since the vast majority of crime victimizations occurred outside the home, where the legality of gun possession might be questionable) to a government surveyor after they've given their name and address. And as he argued in a Politico article in February 2015, experienced surveyors in criminology are sure that "survey respondents underreport (1) crime victimization experiences, (2) gun ownership and (3) their own illegal behavior."

The social science quest for the One True DGU Number is interesting but ultimately irrelevant to those living out those specific stories, who would doubtless be perplexed to hear they shouldn't have the capacity to defend themselves with a gun because an insufficiently impressive number of other citizens had done the same. Even if the facts gleaned from gun social science were unfailingly accurate, that wouldn't make such policy decisions purely scientific.

Could More Guns Mean Less Crime?

The most well-known proponent of the idea that widespread private gun ownership might reduce the rates of violent crime is John Lott, a law and economics professor who has held positions at Yale, UCLA, and the University of Chicago, and who now works as an independent scholar with an organization he runs called the Crime Prevention Research Center. In 1998 Lott published the controversial book More Guns, Less Crime (University of Chicago Press), which was updated with a third edition in 2010. Lott's main argument is that pro-gun policies, such as shall-issue right-to-carry (RTC) laws, tend to lower most crime rates against person and property.

Violent crime has been going down in America in the era when right to carry has spread, but social science is more complicated than simply pointing to two quantities moving in opposite directions.

The most obvious and important fact in modern criminology—the huge decline in crime rates that started a quarter century ago—still lacks anything approaching a universally agreed-upon set of explanations. That fact should help contextualize the picayune and arcane level of argumentation over variables accounted for, specific data sets consulted, and number of different specifications tested when scholars try to buttress or refute Lott's thesis.

The range of contentious issues involved in Lott's techniques were summed up pretty thoroughly in a sympathetic but critical review of the third** edition in Regulation. The economist Stan Liebowitz of the University of Texas at Dallas wrote: "Should county level data or state level data be used? Should all counties (or states) be given equal weight? What control variables should be included in the regression? What violent crime categories should be used? How should counties that have zero crimes in a category, such as murder, be treated? How much time after passage of a law is enough to determine the effect of RTC laws? What is the appropriate time period for the analysis?"

Lott tried to demonstrate that on the county level, violent crime trends showed signs of improvement in counties that had or passed RTC laws compared to counties that had not, among other things checking both mean crime rates and the slope of crime rates before and after RTC passage. He attempted to control for many handfuls of other variables that might affect crime rates—indeed, some researchers accused him of accounting for too many variables, while others slammed him for failing to account for other factors, like conviction rates or length of prison sentences.

Trying to prove Lott wrong quickly became a cottage industry for others interested in the nexus of guns and public safety. The back-and-forths were so extensive that the latest edition of Lott's book is nearly twice as long, with his reactions to his critics.

The U.S. National Research Council (NRC), inspired in part by the Lott debate, assessed the state of the gun controversy in 2004's Firearms and Violence: A Critical Review. The council concluded Lott had not fully proved that RTC laws lowered crime significantly; it also denied that the laws had provably increased crime. "Answers to some of the most pressing questions cannot be addressed with existing data and research methods," study authors Charles F. Wellford, John V. Pepper, and Carol V. Petrie wrote, "because of the limitations of existing data and methods, [existing findings] do not credibly demonstrate a causal relationship between the ownership of firearms and the causes or prevention of criminal violence." That statement is perhaps the most important for people trying to use social science to make gun policy to remember, and there is no strong reason to believe the past decade of research has made it obsolete.

Lott has maintained for years that, even if his critics were right about his positive effects not being robust enough, if you are contemplating for public policy considerations whether expanded RTC is a good, bad, or neutral idea, no one had yet demonstrated that RTC laws made any relevant crime or safety outcome worse.

Then, in 2011, Abhay Aneja, John Donohue, and Alexandria Zhang came out with "The Impact of Right-to-Carry Laws and the NRC Report: Lessons for the Empirical Evaluation of Law and Policy," a paper in the American Law and Economics Review. Working at a very high level of statistical sophistication and running their data through a huge variety of different specifications and assumptions, the authors concluded that "aggravated assault rises when RTC laws are adopted. For every other crime category, there is little or no indication of any consistent RTC impact on crime." (While this kind of social science is always working with subtle attempts to figure out how much more certain quantities might have changed had things been different, it's worth noting that while the number of states with "shall issue" or unrestricted carry permit laws has more than doubled since 1991, aggravated assault rates overall have fallen by 44 percent since 1995.)

The study is suffused with an advanced sense of caution. As the authors write in a 2014 update of that study, "we show how fragile panel data evidence can be, and how a number of issues must be carefully considered when relying on these methods to study politically and socially explosive topics with direct policy implications." They stress "the difficulties in ascertaining the causal effects of legal interventions, and the dangers that exist when policy-makers can simply pick their preferred study from among a wide array of conflicting estimates." And "a wide array of conflicting estimates" is definitely what confronts anyone wading into the social science related to guns and gun laws.

Researchers can and should try to go beyond mere binaries about laws existing or not existing when making subtle assessments of causation. Lott, for example, gets as granular as he can when studying RTC laws, considering not just whether they exist or not, but how easy it is to actually obtain a permit where it's legal to do so. If it's more expensive and time-consuming to get one even in a "shall issue" state, that will likely blunt the law's causal effects at least somewhat.

Along the way, Lott has tried to compile the number of permit holders nationally. He figures the total is 12.8 million, up from 4.6 million as recently as 2007. And now six** states allow so-called "constitutional carry" without a permit, creating a pretty much uncountable body of potential RTC practitioners. We still don't know how many people with gun permits actually carry their weapons, and we have no idea about the end of the causal chain of speculations about how such laws affect crime: what potential criminals believe about how many citizens are carrying guns.

Do 'Common-Sense Gun Laws' Work?

At the top of the list of "common-sense gun safety laws" is expanding background checks beyond the current requirements for federally licensed dealers. The underlying belief here is that the various classes of federally prohibited gun owners, such as felons or those adjudicated mentally ill or known to be drug addicts, should never be able to use "loopholes" such as buying from a private citizen to get a gun (even though the vast majority of all those categories of people would never misuse a weapon).

An April 2015 study by Daniel Webster and three colleagues for the Johns Hopkins Center for Gun Policy and Research earned positive press for claiming that the tougher laws Connecticut passed in 1995 (requiring a background check and a permit for any gun purchase from any source) lowered the state's gun murder rate by 40 percent.

Since Connecticut and most of the rest of the country were all enjoying huge murder reductions in the years after that law went into effect, the researchers couldn't meaningfully compare what happened in Connecticut with what happened in the rest of the country. They needed to compare Connecticut's post-law results to what they think would have happened with gun murders in the state had the law not passed. So they created a statistical model of a "synthetic Connecticut" that was 72 percent comprised of Rhode Island, based in essence on the principle that past results would guarantee future performance, since in the past Rhode Island's murder rates and changes tended to match Connecticut's. Then they compared the two states from 1996-2005. The results? "Connecticut Handgun Licensing Law Associated With 40 Percent Drop in Gun Homicides" blared the Johns Hopkins press release headline.

Rhode Island's murder rate went up unusually after 1997 (the researchers don't speculate on why that might have been), thus creating some "extra" murders (presuming that choices to murder in Rhode Island would have for some reason created a proportional number of choices to murder in Connecticut) that we can credit Connecticut with having evaded thanks to the more stringent gun law.

But what happens when you extend the analytic period beyond the arbitrary cutoff date of 2005? From 2005 to 2012, Connecticut's gun murders per 100,000 people increased 66 percent, from 2.05 to 3.41, while Rhode Island's went down 20 percent, from 1.83 to 1.45. It seems quite premature to take Webster and his team's counterfactual guess about expected murder rates over one 10-year period as establishing any reliable causal knowledge about the effects of tougher gun purchasing laws. Yet that study was used to help buttress a proposed federal law the week it went public, trying to pressure other states into following Connecticut's lead on background checks and permits, given what we now "know" about how life-saving that move had been.

Webster and his colleagues produced a similar but more rigorous study in 2014. It involved actual counts and not assumptions about what might have happened in a counterfactual, and it didn't stop looking at forward data at the most convenient time for its conclusions. This study tried to prove that Missouri's 2007 repeal of its "permit to purchase" law led to a 16 percent increase in murder rates there. Lots of other factors were controlled for, and the numbers indeed showed higher murder rates compared to the U.S. average at the time after the permit law was repealed.

It's tricky to credit the permit-to-purchase repeal with causing that rise, because in the four years prior to eliminating the law, Missouri's murder rates had already gone up 15 percent while the national one had stayed essentially the same. This suggests that unaccounted factors influenced Missouri's rising murder rate both before and after the law changed.

Even if both studies had been flawless, seeing one thing happening in one place over a limited time is usually not sufficient to establish a scientifically valid causal relationship that policy makers can confidently expect to see replicated elsewhere. Aaron Brown, the chief risk manager at AQR Capital Management and a statistician with interest in gun issues, has lamented that the overarching problem with most of these attempts to learn what effect any element of gun prevalence or gun laws has on any real-world outcome is that there simply aren't enough varied data to be sure of anything.

There's another very likely step between "law exists" and "law changes behavior" that most gun social science doesn't, and likely really can't, account for. After Webster's Connecticut study appeared, I asked him: Since you are presuming a strong causal effect from the law's existence, how did you account for how stringently or effectively the law is enforced? If people continued to blithely sell weapons without background checks or permits, that would blunt the notion the law would have such a strong effect on gun murder rates.

Webster's emailed reply: "Virtually no studies of gun control law take enforcement into account because data are lacking and we don't really know the degree to which deterrence (people not wanting to violate the law) is a function of levels of enforcement." Unknowables shadow the causal chain in nearly all social science involving any law's effects on behavior.

Elusive Knowledge

The Duke economist Philip J. Cook put the knowledge problem well in a 2006 Journal of Policy Analysis and Management article. "Policy analysts are trained to critique evaluation evidence, pointing out potential flaws," Cook and co-author Jens Ludwig wrote, "but are perhaps not so well prepared to judge whether the preponderance of the evidence points in one direction or another."

In other words, the most convincing element of any gun study tends to be the part where one scientist is explaining why another one's causal conclusions don't hold up. The parts where they claim strong or definite policy-relevant causal knowledge tend to be much more questionable.

Cook and Ludwig, in their aforementioned 2006 paper "The Social Costs of Gun Ownership," look at this loose link between scientific knowledge and policy differently. They grant that perhaps we're asking more of science than it can give to the policy debate. But that shouldn't stop us from using it to promote more gun law interventions, they maintain. "Suppose [a certain intervention] implies the treatment reduces gun crime by 25% but the p-value on this point estimate is just .15, short of the conventional .05 cutoff," they wrote. "Any academic referee worth her salt would reject a paper submitted for scientific publication that claimed this intervention 'worked.'"

But, Cook and Ludwig wonder, are those scientific standards too rigorous for statecraft? "Would that referee really want to live in a jurisdiction where this evidence persuaded policymakers that they should not adopt the new treatment, but rather stick with the status quo?"

As Harvard's Hemenway explained to me, the confidence intervals of the social sciences in colloquial terms demand a belief that the chances are 19 to 1, or at worst 10 to 1, for you being right about your conclusion before you accept it as provisionally verified. Hemenway also believes, given the good he thinks can come from legal interventions about guns, that we don't need to be that certain we are right for policy work.

But that's easier to accept if you don't value any particular benefits to relatively unrestricted private gun ownership—scientific, constitutional, or just personal. Some researchers, particularly in the public health field, act as if there were no values to balance on the other side of the policy goal of making it harder for people to get guns.

Whether you consider the associations and causations supposedly demonstrated by gun-related social sciences to be proven beyond whatever level of doubt you see as appropriate, applying those stipulated facts to policy questions can never itself be a purely mathematical or scientific process. It's politics all the way down, and that politics is less informed by rigorous and certain knowledge than President Obama thinks.

**Correction: The article originally and incorrectly stated that it was the first edition of Lott's book being reviewed, and that there were only four constitutional carry states.