Government must act to protect workers from AI risks, expert warns

Human oversight of AI being used in Australian workplaces is needed to prevent discrimination against vulnerable workers, a University of Melbourne workplace discrimination expert has warned.

In an analysis of the current state of legal oversight of Artificial Intelligence, published this week in the Melbourne University Law Review,  Associate Professor Alysia Blackham found Australia has few laws protecting the privacy of employees whose information might be gathered and used by their employer’s AI-based systems.

“There is also no requirement for human oversight, even if the AI-based system is going to affect important aspects of people’s lives. Nor do we demand that companies reveal what they are using, and how they are using it, in real offices. No one knows what is going on behind any company’s closed doors,” Associate Professor Blackham said.

The analysis found many human resources departments are buying AI-based programs to help with tasks such as recruitment – choosing which applicants to interview – or for promotion, analysing employees’ work and rating them in terms of productivity, or accuracy. They might also be used to manage performance or in termination.

AI-based programs can go further than human recruiters usually do, for example, by analysing large amounts of social media data, or in using chatbots and other tools to screen and rank candidates, and even to determine salaries.

“It could even mean that there is no specific human involvement behind aspects of decision-making,” Associate Professor Blackham said.

“This is a big concern. Many people assume that AI will be more logical than humans, and less likely to discriminate against groups such as women, culturally diverse communities, people with disabilities and so on.

“But AI systems rely on the data that is used to train them, and this can lead to discriminatory decisions. Take Amazon. It created an AI-based program to sort through candidates’ applications. But the data was from the existing Amazon workforce, where men were hired more often than women. So, the algorithm of the AI system – the set of rules for how it should process data - assumed that meant that women were less desirable, and it reportedly penalised applications with the word ‘women’s’, or the name of any all-women’s colleges.

“Amazon was forced to abandon the program, and this incident is well known. But the internet is still awash with companies trying to sell employers automated recruitment and screening tools.”

^{Melbourne Law School's Associate Professor Alysia Blackham}

Associate Professor Blackham said the Australian Commonwealth public service has also tried to use automated processing for promotion, via a third party, using techniques that included psychometric testing, questionnaires and self-recorded video responses.  The Government’s Merit Protection Commissioner flagged this as deeply problematic, because it sometimes failed to choose the best candidate, and it overturned some of the decisions.

Meanwhile, the Commonwealth Bank was criticised earlier this year for using an app that monitored workers’ location, and how often an employee was at their computer. Managers then followed up by reportedly interrogating suspected cases of absenteeism, ‘half-dayism’ and ‘long-lunchism’, and unproductive workers were allegedly told to apply for leave.

In July, this was followed by a Fair Work Commission decision that found that a sacking based partly on the findings of keystroke-monitoring software was not unfair. In that case, Insurance Australia Group(IAG) ran a ‘cyber audit’ that monitored keystrokes and logins on a work laptop over a three-month period, evaluating daily and weekly productivity. The report found that the employee was not working as required, and this contributed to their dismissal.

Associate Professor Blackham said it can be disturbing to realise that employers can monitor our every movement and every keystroke. But the law offers limited protection.

“Existing employee data in Australia is exempt from privacy legislation. Staff have no control over how anything that is collected about them, from any source, is stored and used. While employers need employee consent to gather new data, in employment, it is nearly impossible to have informed consent to information-gathering – consent is not voluntary if you would lose your job for withholding it.”

Associate Professor Blackham said making companies accountable for their AI use would require changes to data protection and privacy law and equality law, and for unions to be able to demand enforcement of standards in workplace negotiations. Regulatory agencies would also need more power to enforce any laws.

Associate Professor Blackham pointed to the recent turmoil around the sacking and reinstatement of Sam Altman, the CEO of the company Open AI, as evidence of the need for oversight.

“No one outside that company knows what the row is all about, but there are rumours OpenAI was about to develop a system that could evade human control.

“We must demand open reporting on AI development and practices, or it might be that we won’t find out what we need to know until it is too late.”