Rotterdam welfare-fraud risk model

Rotterdam's welfare-fraud risk model was a machine-learning model, originally built with Accenture. The city of Rotterdam used it to score welfare recipients for fraud risk from about 315 inputs, including age, gender, language, and neighbourhood. Its ranked list set who fraud investigators looked at first.

How it was used

Investigators acted on the model's ranked list. The sources describe human review of the ranking as nominal: the ranking drove who was scrutinised.

Investigators recorded each investigation's findings in the recipient's dossier, and the model's risk score was logged there too. They read the dossier again when that person was investigated again.

The sources list human review and an appeal process for this deployment, but describe neither. From 2017 to 2021, officials ranked every benefit recipient in the city by score. Those in the top 10 percent were referred for investigation. The list was one of three ways the city chose whom to interview.

What the reviews found

In 2021 the city's own audit office, the Rekenkamer Rotterdam, published a report on the city's algorithms, Gekleurde technologie. It flagged this model's ethical risks, including proxy discrimination from inputs such as language skill. Proxy discrimination means an input stands in for a trait the law protects from discrimination, such as gender. The city put the model on hold in 2021.

In 2023, journalists at Lighthouse Reports and partners obtained the model and its training data. Their investigation, Suspicion Machines, documented how scores skewed against already vulnerable groups: women, parents, and people with limited Dutch. They found that people who had not passed the Dutch language requirement were almost twice as likely to be flagged.

WIRED and Follow the Money published their own accounts of the model. The Racism and Technology Center reported that it was biased.

Why this case matters

The case file calls Rotterdam its clearest case of an audit acting on a system. The city paused the model after its audit office's 2021 report. In 2023, journalists' access to the model itself documented the bias in detail. Suppliers usually keep their models closed to that kind of outside inspection.

The case also shows a harm that accuracy measures miss. The model decided who got investigated, and an investigation is itself a burden. Asking only whether a flag was correct misses the question of who has to go through an investigation.

The loop to watch

This network assumes a retraining loop. Past dossiers and the outcomes investigators recorded are used to train the next version of the model.

The model then learns from cases its own ranking chose to investigate. So a skew in who gets investigated can return in whom it suspects next.

The sources show the model had training data, which journalists obtained in 2023. Lighthouse Reports' published method says the city retrained the model each year with new data. Follow the Money reports it was trained on 12,700 recipients who had already been investigated.

What the available tools can and cannot address

A failure pathway is a link between two parts of the network, where a mistake made by one part can be passed on to the other. Here a mistake is, for example, a recipient ranked high risk because of their language or neighbourhood, or a wrong finding entered in a dossier. A pathway counts as closed once it passes on only a few mistakes. It need not stop them all.

This case has a budget of 15 units. Explore (No Targets) sets no targets. There, two tools costing 4 units are enough to stop mistakes building on one another across the network. Under Service Targets Only, the same two also meet the service target. One such pair is Mark AI-written records with Escalate checks. Here Mark AI-written records marks the model's scores in the dossiers, so investigators reading them, and the retraining, can weigh them.

Under Service and Safety Targets and All Governance Targets, the targets can also be met. Both levels ask you to close every failure pathway, among other targets. The cheapest combinations under Service and Safety Targets use five tools and cost 11 units. One is Mark AI-written records, Keep prompts neutral, Peer sharing rules, Escalate checks, and Store less data. This model takes no prompts: here Keep prompts neutral acts on how investigators' outcomes are used in retraining.

Under All Governance Targets, the same five meet the targets with Store less data at its stronger setting, for 13 units. At that level, four tools work at reduced strength unless Understand the system is also on. They are Vet connections, Review the riskiest first, Store less data, and Peer sharing rules.

Of the 100,303 combinations within the budget, 169 meet the targets under Service and Safety Targets, and 63 under All Governance Targets.

More is not better here. Every tool at its strongest setting at once closes every failure pathway, but costs 41 units, far over the budget. It also leaves the ranked list no longer clearly helping the investigators' work, so it misses the targets at every level that sets them.