Data Distortion

Lott's numbers don't tell us anything

I reply below to the main criticisms by John Lott -- at least those which I have understood:

Lott doesn't deny that he misleads the reader by neglecting to mention that his plots are fits to the data, because he can't. His graphs are in fact labelled "number of violent crimes" per 100,000 population and I find no statement in his book that the graphs are fits, rather than actual data. In his reply, Lott justifies the use of displaying fits by noting that it is important to show "adjusted" crime rates after other variables (aside from the laws) have been taken into account.

Lott is correct that I was using the first edition of his book when I made the comment about only 10 states changing their right-to-carry laws in the stipulated time period.

Lott claims in his reply that "He [Ehrlich] used data up until 1997, but that is not possible since he limited the sample to only four years after adoption [of the laws]…" Clearly, he is mistaken, since my plots show data extending 10 years before the law's adoption.

My statement about the changes in slope in the various states was based on simple linear fits to the data two years on either side of t=0, without weighting the states by population. However, without doing any statistical analysis whatsoever, a mere glance at the graphs for the 10 states should allow readers to decide for themselves whether the data for the 10 states actually show anything particular happening at time t=0. (The data for robbery can be found plotted in my book or downloaded from the FBI's Web site.)

Lott claims in his reply that his fitting procedure is not biased, because using random data one is notvirtually guaranteed to find a drop or a rise at t=0, as I claimed. Instead, he points out that the random data might show an abrupt change in the slope, not the actual level, at t=0 (e.g., first rising then falling, or first falling then rising). But Lott's correction to my statement actually makes my basic point even stronger, since a decrease in slope is exactly what might be expected if Lott were right. Thus, if his fitting procedure would force random data to show a change in slope at t=0 -- equally often an increase or decrease -- we can't have too much confidence that any observed decrease in slope validates his theory.

It's difficult to find anything about mass murder amusing, but I find Lott's calculation for the greater deterrent effect of easing concealed-carry laws on multiple shootings very humorous. Essentially, he is saying that after concealed carry laws are eased, mass murderers really are more deterred than ordinary murderers, because the chances are much greater that someone in a large group is actually armed. Now, I don't think mass murderers are totally irrational. But I find this type of probability calculation more revealing of Lott's thinking than that of mass murderers, some of whom I imagine would relish the idea of going out in a blaze of glory, in case someone in the group were armed. ("Suicide by police" seems to be a fairly common act by some psychos.)

In Lott's rebuttal on this same issue he fails to address the other inconsistency in his results: How could the laws act in reverse time, causing a big spurt of mass shootings the year before the laws were enacted? He also neglects to answer my question on how his analysis can show the murder rate dropping immediately after the laws are passed, but the aggravated assault rate not starting its drop until four years later.

Lott is right in pointing out that the omitted variables would need to change systematically in a way correlated with the dates of passing the laws. But given that the laws (according to him) account for such a tiny fraction of the change in crime rates, and given an extremely long list of possible variables, it seems likely that some of them could fit the bill. If Lott's claim that he really has accounted for all the key variables that affect violent crime rates were correct, then he really should be able to predict how the crime rates will change in the future in each state, based on all these variables. Moreover, if his predictions fail to be borne out in any state it would show that he has left out some factor. (We are all used to hearing about why the stock market did what it did on any given day, after the fact. But the failure to make such accurate predictions ahead of time tells us that maybe we really don't fully understand all the variables that make the market do what it does, any more than we understand the variation in crime rates.)

I am not alone in questioning Lott's statistical analysis – see, for example, work by Daniel Webster, Jens Ludwig, Daniel Black, and Daniel Nagin. Lott notes that his F-test is the appropriate one to answer the question of whether there was a statistically significant change in the slope in crime rates at t=0. I don't dispute that the change in the slope of crime rates may be statistically significant at t=0. After all, there might have been a real change at that point in time for reasons unrelated to the laws.

However, I claim that the slope will probably also be found to change by statistically significant amounts at most other years as well, and that would show that there's nothing special happening at t=0, the year the laws were passed. The real test that it was the liberalized gun laws that made the difference is that a statistically significant change in slope was found at t=0 and only at t=0.

To see this basic flaw in Lott's statistical analysis, let's imagine that some lunatic has a theory that the NASDAQ drops every full moon. Presumably, according to Lott, the way to test this theory would be to do a linear regression involving as many extraneous variables as we can think of that might affect the NASDAQ -- and not to worry too much that we may not have gotten them all. Then using the regression, we need to see if the NASDAQ had a statistically significant drop on days when the moon was full. It very well might show a statistically significant drop on those days. Why not? However, I expect that the NASDAQ would also show drops (and rises) having comparable statistical significance for other lunar phases as well -- thereby proving exactly nothing.

Prof. Lott, wouldn't you agree that a finding that there was a statistically significant change in the crime rates at years before t=0 would invalidate your results? Will you tell us what your analysis shows for the statistical significance of changes in slope at years other than t=0?