Transparency in Sentencing

The QSIDE Institute has released a comprehensive dataset of federal court sentencing data, cobbled together from a number of sources, including PACER. The dataset has over 570,000 records, each corresponding to a single sentencing, from 2001-2018. The project represents a phenomenal investment, and it should help make sentencing more transparent. In particular, because the dataset includes judge identifiers as well as defendant characteristics, it may help address questions such as whether particular judges discriminate against minorities in sentencing.

For example, Chad Topaz has tweeted a graph from a not-yet-published analysis that, he reports, indicates that "30+ judges display … statistically significantly different sentencing behavior by defendant race." Many of these judges are more likely to give a below-Guidelines sentence to a white defendant than to a "minoritized" defendant, and many also are more likely to give an above-Guidelines sentence to a minoritized defendant than a white defendant. At this stage, the judges are not named, though the underlying data is public.

More nuanced analysis is promised, from David Abrams, Sonja Starr, and Crystal Yang, and I look forward to it. [Correction: Topaz was indicating only that these scholars would be capable of offering nuanced analysis. I agree and apologize for the misreading.] It will be difficult even from this data source, however, to make conclusive determination of whether minority defendants are treated worse in sentencing, all else equal. Black defendants may be systematically different from white defendants in ways that the data cannot capture, so overall racial disparities may be suggestive but not conclusive. Still, large differences in sentencing controlling for available information, as have been found in past studies on smaller datasets, may be highly suggestive of discrimination. The provision of more data on this important question is a valuable contribution.

We may be able to obtain somewhat better information about variations in judicial behavior. After all, cases are generally randomized to judges. If Judge A and Judge B are in the same district and over a large number of cases, Judge A's differential treatment of white vs. minority defendants relative to Judge B's is statistically significant, then we can infer that one judge is more generous to white defendants than the other. But Topaz identifies "VERY IMPORTANT CAVEATS" (caps in original). Just because a disparity exists does not necessarily mean that a judge is biased. As just one possibility that Topaz notes, the judges may differ in whether they vary sentences based on whether a defendant has dependents, a variable that itself may be correlated with race. The fact that a dataset includes many control variables does not exclude the possibility that it omits other important controls, and even where a dataset includes a control, if that control is only a proxy for some underlying variable, the regression only partly controls for that underlying variable.

Once data about how judges differ on the basis of race are publicly released, that may place pressure on judges to change their sentencing patterns so that they treat different races equally. One might argue that this is harmful, if it means that judges stop taking into account race-neutral variables that they genuinely care about in sentencing. But if a particular judge is an outlier in this regard, then most judges apparently agree that this consideration should not factor into sentencing. Topaz is right that judges who are systematically harsher on defendants of a particular race may not be biased (because other unidentified factors may explain their decisionmaking), but they are at least outliers. Transparency in sentencing may, like the Sentencing Guidelines, serve as a mechanism for reducing sentencing disparity.

A more worrisome possibility is that with the release of such data, judges start directly taking into account race, albeit not admitting that they are doing so. Judges might keep internal track of their record of sentencing defendants of different races. A judge who sees an apparent bias on a public dashboard, even a statistically insignificant one, might shade decisionmaking in the direction of reducing bias. (I assume that judges would not want to appear either biased for or against minority defendants, though it is theoretically possible that some judges would want to appear biased in one direction or the other.) Perhaps in the future, we might determine that there is less difference among judges in bias measures than one would expect from random chance, suggesting that judges are ignoring relevant considerations in order to appear unbiased.

This highlights a more general point about transparency: If greater amounts of data allow people to be scored based on some considerations but not others, then they will be relatively more likely focus on those considerations and ignore the others. Thus, if transparency leads to seemingly better performance on one variable, it may be at the expense of other variables. More alarmingly, if someone wants to make it appear that a variable does not influence the person's decisionmaking, the person counterintuitively may allow the variable to play an explicit part in decisionmaking, to counter any random fluctuations that otherwise might incorrectly be interpreted as bias.

Still, my instinct is that the benefits of transparency greatly outweigh these costs. There is an appropriate national focus on race in the criminal justice system. Unless one has a very strong prior that race does not play any role in the criminal justice system and that seeming patterns reflect spurious correlations, one should favor better data. And if that data indicates that judges are outliers in one direction or another, it seems to me more likely than not that the scrutiny of those judges will be more likely to make sentencing genuinely consistent and fair than to achieve a false consistency.