System Errors and Due Process

How can government agencies better safeguard procedural due process rights?

As Supreme Court Justice Felix Frankfurter put it: "The history of American freedom is, in no small measure, the history of procedure." The scale of threats to due process, however, has greatly expanded. And so have the opportunities to correct uses of wholly error-prone systems by government, in ways that harm our liberty and property rights.

In Minnesota, a recent federal lawsuit alleges that a private insurer illegally denied "elderly patients care owed to them under Medicare Advantage Plans" by deploying an AI model known to have a 90% error rate, to override determinations by physicians. The tool is proprietary, and the company denied requests from patients or physicians for more information; I am aware of no independent research evaluating its performance. If the allegations are correct, then the model, used in many states, could have resulted in errors in millions of cases. (Federal regulations, made effective in 2024, now forbid exclusive reliance on AI to make such determinations.)

Another significant source of error involves not an automated system operating alone, but the way human decision-makers rely on something automated. As NIST has explained, in the context of facial recognition technology, "what matters then is the human response," because the human police officer will review the candidate photos that the system pulls to decide whether any of them is a suspect. The system may score the images, or call the level of match "strong," using unclear criteria, and, as the National Academy of Sciences concluded in a landmark 2024 report, such interpretations "can prejudice or bias human review of images." The evidence on the interaction of these systems and the people that use them is limited, but what research has been done suggests that people can be very bad at recognizing a face from a series of images. Even in a setting when the task was to examine high-quality passport photos, participants made errors 50 percent of the time. Eyewitness misidentifications are a leading cause of wrongful convictions, and it is well understood that suggestive police lineups can alter the memory of an eyewitness. And we know that false arrests can and have resulted from the use of untested facial recognition systems. Except in a handful of states that have passed some regulations, the roll-out of facial recognition has occurred freely.

Another source of error may lie buried in the code used to program a software tool that the government relies upon. In another noteworthy example, in People v. Collins, a New York trial judge found the secrecy behind a government program used to interpret complex DNA mixtures, called the Forensic Statistical Tool (FST), to be highly problematic: "The fact that FST software is not open to the public, or to defense counsel, is the basis of a more general objection… [T]he FST is, as a result, truly a 'black box'—a program that cannot be used by defense experts with theories of the case different from the prosecution's." The FST was a "black box" computer program only because the government refused to disclose the underlying code. When a defendant made objections, however, a federal judge ordered the code to be disclosed. At that point, outside experts reviewed it and found serious flaws. And the New York Medical Examiner's Office stopped using the error-ridden software.

Our courts are only slowing coming to grips with the ways that modern technology can increase the chances of an error. In Herring v. United States, decided in 2009, the Supreme Court Justices examined this issue in the context of basic databases that local governments maintain to track criminal cases. A defendant, Herring, was arrested based on a warrant. This was an error; the warrant had actually been recalled, or canceled, several months earlier, but the government database had not been updated. The majority of the Justices decided that, because police acted in "good faith" when they arrested Herring, and the mistake in the database was not "systematic error," Herrings's conviction should not be reversed. Could the Justices be sure that these errors were not systematic? As Supreme Court Justice Ruth Bader Ginsburg put it in dissent, although databases "form the nervous system of contemporary criminal justice operations … [t]he risk of error stemming from these databases is not slim."

The best way to minimize serious errors is to try using basic quality controls to look for errors, cure them, and make improvements to the process. Even rudimentary error checks are often lacking for government processes that deeply matter to people's lives and their rights. Shortly after the Supreme Court's landmark procedural due process ruling in Goldberg v. Kelley was decided, Yale scholar Jerry Mashaw described the "management side of due process," and called on government agencies to adopt a "quality assurance system," since fair hearings would not be enough to assure accuracy in decision-making. Due process requires good management, Mashaw argued, and not just individual hearing rights.

More recently, David Ames, Cassandra Handan-Nader, Daniel E. Ho, and David Marcus, have conducted important work examining mass errors in mass adjudication by agencies, focusing on the Board of Veteran's Appeals. Due process has little meaning of errors are rampant, and if there is no sound system for detecting and correcting them. They argue that: "Goldberg's original premise of decisional accuracy requires a hybrid of external intervention, stakeholder oversight, and internal agency management." Instead, we have seen, as I discussed in a prior post, the deployment of new AI systems that raise the specter of still more rapid and large-scale errors by administrative agencies. This is important for government officials too. It harms the efficiency of their work if it is unfair and error prone, as David Super has explored; due process does not necessarily involve a trade-off between efficiency and individual rights.

Actual testing and error rates should be taken far more seriously as a central part of what it means for due process to protect people against erroneous deprivations of life, liberty, and property. In the courts, standards of review make it very difficult to correct errors on appeal; this is especially true in criminal cases, where I have studied the post-conviction litigation of people who were eventually exonerated by DNA testing. Judges do not normally act like systems administrators; instead, apart from their role in aggregate litigation, they handle cases one at a time. Yet, even large, well-resourced agencies that process vast numbers of cases do not have incentives to audit themselves. Our custodians of due process should be required to test for and correct serious errors, and absent that, we should be proactive in legislation, to impose new due process safeguards requiring government officials to use quality control systems to minimize errors.