No, Trump Rallies Didn't Increase Hate Crimes by 226 Percent

In fact, they didn’t have any detectable impact at all.

In the aftermath of August's mass shooting in El Paso, a cursory glance at the news might have left you with the impression that there were two villains that day: the white supremacist who viciously killed 22 people and wounded 24 more, and President Donald Trump, who supposedly incited him. 

"The president cannot be absolved of responsibility for inciting the hatreds that led to El Paso," read a New York Times piece. Democratic presidential contenders echoed this sentiment, with Beto O'Rourke saying Trump's rhetoric "has a lot to do with" the shooting and Kamala Harris alleging that Trump was "tweeting out the ammunition" used by the El Paso shooter.

Blaming the words of controversial politicians for the acts of terrorists and lunatics without hard evidence is not new. However, a recent academic paper, reported on by numerous outlets before it went through the peer-review process, suggests that Trump actually is to blame. 

Studying the effects of Trump's many campaign rallies on reported hate incidents, three professors at the University of North Texas and Texas A&M—Ayal Feinberg, Regina Branton, and Valerie Martinez-Ebers—claim that Trump rallies are associated with a 226 percent increase in such incidents. 

Naturally, their study went viral. Vox, The Washington Post, the San Francisco Chronicle, and CNN all published articles reporting that Trump's words are so bad that exposure to them leads to a wave of hate crimes.  

Once again, Democratic politicians piled on. "Your language creates a climate which emboldens violent extremists," wrote Sen. Bernie Sanders (I–Vt.) in a social media post. "Your rhetoric is directly and indirectly inciting hate, Mr. President," tweeted Rep. Ilhan Omar (D–Minn.).

Is that where the story ends? Not quite. Using the same data and statistical procedures as Feinberg et al., we replicated their study's headline result. Since we did not have access to the original paper's data and code, this involved collecting each of the variables mentioned in the original paper, and then independently performing the same analysis. Wherever possible, we copied the decisions that are mentioned in the original paper. Our headline results were very close to those reported in the original paper. (You can see our work at GitHub.)

Using additional data we collected, we also analyzed the effect of Hillary Clinton's campaign rallies using the identical statistical framework. The ostensible finding: Clinton rallies contribute to an even greater increase in hate incidents than Trump rallies.  

This should be enough to give any reader pause. The implied reasoning of those who cited the initial study was that Trump's caustic and seemingly racist rhetoric contributed to a crueler, more discriminatory climate, ripe for hate crimes. If this interpretation is correct, why did Clinton inspire as many, if not more, hate incidents as Trump did? Did calling millions of Americans "deplorables" promote violence?

Probably not. Both of these results rely on comparing counties with rallies to other counties without them. This produces a glaring problem. Politicians tend to hold political rallies near where large numbers of people live. And in places with more people, the raw number of crimes is generally mechanically higher. Simply put, no one should be surprised that Orange County, California (population 3.19 million) was home to both more reported hate incidents (5) and Trump rallies (2) than Orange County, Indiana (population 19,840, which had zero of each). 

Nor is it sensible to interpret that one of these differences (hate crimes) is caused by the other (political rallies). Indeed, adding a simple statistical control for county population to the original analysis causes the estimated effect of Trump rallies on reported hate incidents to become statistically indistinguishable from zero. The study is wrong, and yet journalists ran with it anyway. 

How could this happen?

Even if researchers attempt to be unbiased, it is easy for ideology to interfere with the practice of the scientific method. Empirical work requires numerous small assumptions and choices, often without obvious right and wrong options, that collectively affect the findings. The hypotheses that researchers choose to test often reflect their beliefs, and when initial statistical findings do not match the researcher's gut intuition, it is easy to tweak these choices until the analysis "works." By comparison, when a result "feels right," it is easy to conclude the analysis with minimal further checks. In short, falling prey to confirmation bias is easy. Attempting to find errors in a result that, deep down, you want to be true, is hard. Furthermore, the ideological imbalance of academia—where liberals outnumber conservatives six to one—can worsen this. While claims deemed conservative may receive much scrutiny, those that comport with liberal sensibilities are more likely to go unscrutinized.

In principle, this need not have much impact outside academia. A neutral press, acting as a gatekeeper, need not report unquestioningly about every unpublished study. However, like academics, journalists as a profession are overwhelmingly liberal, with four times as many reporters identifying as Democrats than as Republicans. Given how little scrutiny was required to reveal the flaws in the thesis that Trump rallies cause hate incidents, one cannot help but wonder whether its viral status was aided by journalists predisposed to believe its message. Would a study claiming Clinton rallies caused hate crimes to increase by 226 percent have been seized on equally enthusiastically? We are skeptical. Because of this, we have some sympathy for the authors of this study—such errors are typically found during peer review and quietly corrected before publication. Yet, these errors have profoundly different implications when they result in widespread, extraordinary-but-false claims in the popular media. 

Many of Trump's fiercest critics have proclaimed themselves to be staunch defenders of science in the face of a supposedly rising tide of anti-scientific sentiment. By rushing to promote academic results that are most in line with their own preconceived notions, Trump's critics risk committing the very error they decry. It is bad for America when Trump expresses falsehoods or unnecessarily inflammatory rhetoric. News organizations of record shouldn't do it either.