The unregulated use of algorithms by the Dutch tax office to create risk profiles of potential benefit fraudsters led to both discrimination and racial profiling, according to a damning report by Amnesty International.

The English language report, Xenophobic Machines, which is being presented at a side meeting to the UN General Assembly in New York on Tuesday, said the ‘Dutch childcare benefit scandal is an urgent wake-up call to ban racist algorithms’.

‘Thousands of lives were ruined by a disgraceful process which included a xenophobic algorithm based on racial profiling,’ said Merel Koning, senior advisor on technology and human rights at Amnesty International. ‘The Dutch authorities risk repeating these catastrophic mistakes as human rights protections are still lacking in the use of algorithmic systems.’

The Amnesty researchers found ‘Dutch citizenship: yes/no’ was used as a category in the childcare benefit risk classification model and this, the organisation says, meant non-Dutch nationalities received higher risk scores and amounted to racial profiling.

‘The use of nationality in the risk classification model reveals the assumptions held by the designer, developer and/or user of the system that people of certain nationalities would be more likely to commit fraud or crime,’ the report said.

‘It is also indicative of the tax authorities’ perception that there is a link between race/ethnicity and crime, as well as an acceptance of the practice of generalizing the behaviour of some individuals to all others who are perceived to share the same race or ethnicity.’

Impact

The researchers point out that the Dutch tax authorities did not assess the human rights risks and impacts prior to using the risk classification model and did not take any steps to prevent discrimination.

‘The lack of transparency and accountability of the tax authorities drove the victims of the childcare benefits scandal into a vortex of bureaucratic and opaque legal procedures, with no effective remedies available to the parents and caregivers that were discriminated against with the risk classification model,’ Amnesty said. 

The report also notes that while the Dutch government publicly disapproves of racial profiling, it continues to allow the use of ethnicity and other discrimination risk factors as a basis for suspicion and for decision-making in law enforcement.

Court case

In September, for example, a Dutch lower court ruled that border police can continue selecting people for extra identity checks on the basis of their ethnicity because this does not amount to discrimination.

‘The mere fact that ethnicity plays a role… is not by definition discriminatory and cannot therefore constitute grounds for a general ban,’ the judge said.

The ruling led junior justice minister Ankie Broekers-Knol to confirm that border police work with risk profiles based on information from the security services, as well as other risk indicators. These could include race and the travel companions as well as behaviour and travel routes, she said.

Ombudsman

National ombudsman Reinier van Zutphen said in March the government and official agencies must take ethnic profiling far more seriously, arguing that few people make complaints about being ethnically profiled because they don’t have sufficient trust in the authorities to sort it out.

‘We know it happens in the Netherlands,’ he said. This makes it all the more important that government bodies look into complaints properly rather than saying ‘what makes you think that’.’

The ombudsman looked into 159 cases in which people said they had been the victims of ethnic profiling, of whom three-quarters had not made a formal complaint.