Welcoming Our New Algorithmic Overlords?
Algocracy and the moral and political legitimacy of government decision-making
Algorithms are everywhere, and in most ways they make our lives better. In the simplest terms, algorithms are procedures or formulas aimed at solving problems. Implemented on computers, they sift through big databases to reveal compatible lovers, products that please, faster commutes, news of interest, stocks to buy, and answers to queries.
Dud dates or boring book recommendations are no big deal. But John Danaher, a lecturer in the law school at the National University of Ireland, warns that algorithmic decision-making takes on a very different character when it guides government monitoring and enforcement efforts. Danaher worries that encroaching algorithmic governance, what he calls "algocracy," could "create problems for the moral or political legitimacy of our public decision-making processes."
Given algorithms' successes in the private sector, it is not surprising that government agencies are also implementing algorithmic strategies. The Social Security Administration uses algorithms to aid its agents in evaluating benefits claims; the Internal Revenue Service uses them to select taxpayers for audit; the Food and Drug Administration uses them to study patterns of foodborne illness; the Securities and Exchange Commission uses them to detect trading misconduct; and local police departments employ algorithmic insights to predict both the emergence of crime hotspots and which persons are more likely to be involved in criminal activities.
Most commonly, algorithms are rule-based systems constructed by programmers to make automated decisions. Because each rule is explicit, it is possible to understand how and why the algorithm produces its outputs, although the continual addition of rules and exceptions over time can make keeping track of what the system is doing ever more difficult.
Alternatively, various machine-learning algorithms are being deployed as increasingly effective techniques for dealing with the growing flood and complexity of data. Broadly speaking, machine learning is a type of artificial intelligence that gives computers the ability to learn without being explicitly programmed. Such learning algorithms are generally trained to organize and extract information from being exposed to relevant data sets. It is often hard to discern exactly how the algorithm is devising the rules from which it makes predictions.
While machine learning offers great efficiencies in digesting data, the answers supplied by learning algorithms can be badly skewed. In a recent New York Times op-ed, titled "Artificial Intelligence's White Guy Problem," Kate Crawford, a researcher at Microsoft who serves as co-chairwoman of the White House Symposium on Society and Artificial Intelligence, cites several such instances. For example, in 2015 Google Photo's facial recognition app tagged snapshots of a couple of black guys as "gorillas." Back in 2010, Nikon's camera software misread images of Asian people as blinking.
"This is fundamentally a data problem. Algorithms learn by being fed certain images," notes Crawford. "If a system is trained on photos of people who are overwhelmingly white, it will have a harder time recognizing nonwhite faces." As embarrassing as the photo recognition problems were for Google and Nikon, algorithmic misfires can have much direr consequences when used to guide government decision making. It does not take too much imagination to worry about the civil liberties implications of the development of algorithms that purport to identify would-be terrorists before they can act.
In her op/ed, Crawford cites the results of a recent investigation by ProPublica into how the COMPAS recidivism risk assessment system evaluates the likelihood that a criminal defendant will re-offend. Judges often take into consideration COMPAS risk scores when making sentencing decisions. Crawford notes that the software is "twice as likely to mistakenly flag black defendants as being at a higher risk of committing future crimes. It was also twice as likely to incorrectly flag white defendants as low risk."
In Wisconsin, Eric Loomis has filed a legal challenge against his sentencing judge's reliance on his COMPAS score. Based on his risk score, Loomis was told that he was "high risk" and was consequently sentenced to six years in prison for eluding police. His attorney is arguing that he should be able to get access to the proprietary algorithm and make arguments with regard to its validity.
Lots of police departments are now using predictive policing programs such as PredPol. According the company, PredPol's algorithm uses only three data points—past type, place, and time of crime—to pinpoint the times and places where a day's crimes are most likely to occur. Crawford worries that predictive policing could become a self-fulfilling prophecy in poor and minority neighborhoods, in the sense that more policing could lead to more arrests, which in turn predict the need for more policing, and so on. "Predictive programs are only as good as the data they are trained on, and that data has a complex history," she warns.
Cities plagued by violence are turning to predictive policing programs in an effort to identify in advance citizens who are likely to commit or be the victims of violent crimes. For example, Chicago police have been applying an algorithm created by researchers at the Illinois Institute of Technology that uses 10 variables, including whether an individual's criminal "trend line" is increasing, whether he has been shot before, and whether he has been arrested on weapons charges. In order to avoid racial bias, the program excludes consideration of race, gender, ethnicity, and geography.
The program has identified some 1,400 Chicago residents who are highly likely to shoot or be shot. As The New York Times reported in May, police warn those highest on the list that they are being closely monitored and offer social services to those who want to get away from the violence. The algorithm's output is fairly accurate: 70 percent of Chicago residents shot and 80 percent arrested in connection with shootings in 2016 were on the list. Since the algorithm is proprietary, there is no way for people to challenge being on the list.
Can algocracy be tamed? Danaher argues that both resistance and accommodation are futile. Resistance will falter because the efficiencies and convenience of algorithmic decision-making processes will, for most people, outweigh their downsides. Trying to accommodate algorithmic decision-making to meaningful human oversight and control will also founder. Why? Because such systems will become increasingly opaque to their human creators. As machine-learning algorithms trained on larger and larger datasets generate ever more complex rules, they become less and less interpretable by human beings. Users will see and experience their outputs without understanding how such systems come to their conclusions.
Officials may still be in the decision-making loop in the sense that they could reject or modify the machines' determinations before they are implemented. But they may be reluctant to interfere with algorithmic assessments due to a lack of confidence in their own understanding of the issues involved. How likely is a judge to let a criminal defendant with a high-risk recidivism score go free?
"We may be on the cusp of creating a governance system which severely constrains and limits the opportunities for human engagement, without any readily available solution," Danaher concludes. Even if algocracy achieves significant gains and benefits, he warns, "we need to be sure we can live with the tradeoff."