If computers took over the world, what would happen to Medicaid? I recently sat in on a presentation given by visiting professor Danielle Citron who argued that the computer systems used by governmental agencies that are increasingly automating their due process decisions. This mechanism is ill-suited to protect individuals from arbitrary agency action.
“I am currently developing a new model of procedural regularity and policy-making that can operate when pivotal government decisions are made by automated systems and the programmers who design them,” said Citron. “Computer systems are designed to make decisions about individual rights including the termination of Medicaid benefits and excluding certain passengers from flying on commercial airliners.”
The decisions these systems make, therefore, have potentially devastating implications. Take, for instance, CBMS, Colorado’s public benefits agency, whose IT employees and an outside vendor crafted a rulemaking algorithm that translated food stamp eligibility policies into code. “The automated program required eligibility workers to ask individuals seeking food stamps and other cash benefits whether they were ‘beggars,’ even though neither federal nor state law required an answer to that question,” explained Citron.
Among the reasons Citron cited for automation problems were the distortion of policy language into code and the computer’s inability to understand the complexities of the human language. “My piece aims to shape a response to these problems,” she explained. “For one thing, agencies should begin insisting on audit trails… we must also tackle the automation bias.” Citing the CMBS case, Citron explained that a programmer’s political views or bias may have influenced the wording of questions that discriminated against certain groups. “It is possible that a programmer could have encoded consequences to certain answers to that question that would reflect the programmer’s political views.”
Other ways Citron recommended combating automation bias are instituting norms of transparency among programmers and policymakers and conducting thorough tests on the systems for legal compliance. “There’s a gap in legal programming, and lawyers really ought to lend an eye to this problem,” commented one attendee.
Sounds like this might be an interesting consideration for cyber-savvy law students!
