Artificial Justice: the human cost of algorithms

Using computerised risk scores to make decisions about people can cause them deep and unjustified harm, warns a Melbourne legal expert in a new book.

“Algorithms are being used to send people to prison, or deny them medical treatment because they experienced poverty, unstable housing, abuse or neglect as a child,” says Associate Professor Tatiana Dancy of Melbourne Law School, author of the new book Artificial Justice.

A predictive algorithm is a set of rules based on observations about large groups of people. She warns that if that data is based on existing inequities, the results will also probably be discriminatory: “For instance, Facebook ads for checkout assistants were shown to an audience of 85 percent women, and Google showed ads for high-paying roles more often to men than women.”

In a case close to home, NSW police used a computerised risk assessment tool to help them target offenders with relentless surveillance, and interventions like stop and search, to try to prevent them re-offending. The youngest “target” was ten. NSW last year abandoned this Suspect Target Management Plan after an inquiry found it to be unjust and focused disproportionately on Aboriginal people.

But A/Professor Tatiana Dancy says that there are also problems beyond inequality.

For instance, an algorithm that predicts future criminal offending might base its prediction on characteristics such as a person’s poverty, their neighbourhood and experiences of childhood abuse or neglect. This can go into a computer program that predicts risk and helps judges decide whether to send an offender to jail, as happens in the UK (OASys) and the US (COMPAS).

“The effect can be that a judge imprisons a first offender because they had a history of childhood abuse. That’s clearly deeply concerning,” she says.

“It means that person is being penalised not because of something they chose to do, but because of something that happened to them, and over which they had no influence.”

Similar problems arise in healthcare, says A/Professor Dancy. The Opioid Risk Tool (ORT) has been used in Australia and elsewhere to decide whether to prescribe strong painkillers, and aims to predict whether the patient is likely to become addicted to the medication. Those predictions are based on factors including whether female patients were sexually abused as children.

The result was that many more women than men were refused pain relief – because of their childhood trauma.

“Even if algorithms could do (and be proven to do) a better and more equal job of making some predictions than people, we should be deeply concerned about the idea that we can send someone to prison, or deny them medical treatment, because they experienced abuse or neglect as a child,” says A/Professor Dancy.

“We all want the ability to shape our own lives. We want to be able to avoid attracting the special attention of police, or a spell behind bars, by staying within law. And we want to be able to receive suitable medications by behaving responsibly.”

A/Professor Dancy argues that, when we send someone to prison because of harm they suffered as a child, we deny these choices. “We treat people as if they were always going to be wrongdoers – as if their fate were sealed from the beginning.”

Artificial Justice, by Tatiana Dancy, published by Oxford University Press.

Media enquiries: Karen Kissane | 0448 286 084 number | kkissane@unimelb.edu.au