A growing number of decisions that affect our lives are being made by machines. A new study highlights automated decision-making systems used by the government of the District of Columbia, and concludes those decisions are affecting fairness and equity.
The Electronic Privacy Information Center’s Screened and Scored in DC report, released Tuesday, looks at some of the automated decision making systems — or ADM — used by the D.C. government, but often provided by third-party companies.
The study says automated decisions have wide implications, affecting economic opportunity, health, education and housing.
In the criminal justice system, an ADM system used during pretrial hearings creates an output of how likely a defendant is to get arrested again, or fail to appear at trial. Information about a defendant’s criminal history, employment status and demographic information are fed into an ADM tool called a “risk assessment instrument.”
According to the study, “The risk assessment instrument automatically applies different weights to each piece of information and aggregates a risk score.”
For housing, the D.C. Housing Authority uses an algorithm that screens applicants for their criminal histories and likelihood of making rent payments on time. Other algorithms assist case workers in determining which applicant is most in need, and who gets housing first.
In monitoring applications for food stamps, the D.C. Department of Human Services uses an algorithm to generate a fraud risk score.
The automated decisions are made without much oversight, according to the report: “Because people impacted by automated decision-making systems are rarely given much information about how they work, it is hard to determine whether these systems are accurate.”
Without transparency, the study says, automated decision making systems make decisions that are hard to challenge — unlike a decision by a judge, which produces a paper trail, and opportunities for appeal.
“You have a right to due process when the government makes a decision about your fundamental rights, such as your eligibility for public benefits. Due process ensures that the government cannot arbitrarily, unfairly, and/or inaccurately deprive you of your rights to important resources such as unemployment insurance, nutrition assistance, healthcare, or disability benefits.”
The study says automated decision-making perpetuates algorithmic harm, in ways that aren’t equitable.
As an example, the report points to the Metropolitan Police Department’s use of ShotSpotter technology. ShotSpotter uses a series of publicly installed acoustic sensors to “hear” gunshots, estimate via algorithm where the noises are coming from, and send a police response to the location.
ShotSpotter says its coverage areas are determined by police using objective, historical data on shootings and homicides to help identify areas most impacted by gun violence. However the report said “police place ShotSpotter sensors almost exclusively in Black and brown neighborhoods.” The report cites studies in other cities showing ShotSpotter alerts often yield no evidence of a gun-related crime.
“False ShotSpotter alerts send police — expecting an armed suspect — into communities whose members are already more likely to be harmed or killed by officers. ShotSpotter has the potential to contribute to, rather than alleviate, violence,” according to the report.
The report offers resources for residents to ensure their rights are protected, as well as recommendations to organizations and policymakers.
