Amsterdam Smart Check

Slimme Check (Smart Check) was a machine-learning model the City of Amsterdam built to screen new applications for social assistance. It scored each application and labeled some investigation-worthy, meaning worth investigating for possible fraud. The label was advice to the enforcement staff who reviewed it, never a decision.

How it worked

The city began designing Slimme Check in 2019, and the effort ran about five years. It was an Explainable Boosting Machine, a kind of machine-learning model built so people can see how each input moves the score.

It scored each application on 15 behavioral and administrative inputs. They came from the city's personal-records database, with citizen ID, address, and housing situation, and from Income Services systems, with prior applications, assets, and income. The city deliberately left out gender, nationality, and age, and inputs known to stand in for them, such as postal code. Slimme Check learned from about 3,400 past enforcement investigations.

A score above a set cut-off produced an advisory investigation-worthy label. Three people in separate roles then took over. An enforcement employee reviewed each label. A separate investigator carried out any detailed investigation, with powers that include access to bank records and home visits. A different Income Services official made the final decision on the benefit.

Staff were trained, and given written instructions, not to rely on the label too much. The Dutch national algorithm register calls this meaningful human intervention, not automated decision-making.

The safeguards before the pilot

The case file calls these safeguards as complete as the period offered. The city carried out a data-protection impact assessment, and an IAMA, a Dutch assessment of an algorithm's effects on human rights. It also carried out an AI impact assessment and a quick security scan against the Dutch government's information-security baseline. The city's data protection officer and its Personal Data Commission reviewed the design.

The city commissioned a review from Deloitte, about 35,000 euros by the city's own estimate, and consulted the University of Amsterdam's Civic AI Lab. It published entries in both the Amsterdam and the Dutch national algorithm registers.

It also convened a citizen Participation Council of up to 15 benefit recipients and interest-group representatives. The council was briefed in depth in late 2021. MIT Technology Review dates the first briefing to the fall of 2021, and the city's research platform says the end of 2021.

In March 2022, the council concluded the project should be stopped. It cited variables such as year of birth that could discriminate, disproportionate data processing, and use of an algorithm without transparency or democratic control. The city removed some variables and strengthened data safeguards. It overrode the core advice to stop and went ahead with a pilot.

The bias test and the fix

A wrongly flagged application is one labeled investigation-worthy when there was nothing to find. Internal testing in May 2022 found the model wrongly flagged non-Dutch applicants about 30 percent more often. It wrongly flagged non-Western applicants nearly twice as often, and men about 14 percent more often.

The team then reweighted the training data, changing how much each past case counted. On past data, this made wrongful flags roughly equal between Western and non-Western applicants.

What the pilot showed

In 2023, Slimme Check ran in a pilot on nearly 1,600 real applications. The city's research platform gives the window as April to August 2023. The investigation analyzed pilot data from June to August 2023.

The fairness did not carry over. Dutch nationals and women were now more likely to be wrongly flagged. In the investigation's analysis, women were about 22 percent more likely. A new bias against applicants with children appeared, which the city had not publicized. Slimme Check flagged more applications than the caseworker process, not fewer. It was no better than caseworkers at picking out the applications that did warrant investigation. The city's internal claim before the pilot, an accuracy edge of about 20 percent over caseworkers, did not hold.

The city halted the project. The Dutch national algorithm register records the deployment ending in September 2023 and lists Slimme Check as out of use. In late November 2023, the alderman responsible, a member of the city's executive, announced the end publicly. He said he could not have justified continuing a pilot that showed the algorithm contained substantial bias.

Where the figures come from

The group figures and the flag counts are not city publications. Journalists from MIT Technology Review, Lighthouse Reports, and Trouw computed them, with Pulitzer Center support. They worked from confusion matrices the city provided: tables counting correct and wrong flags for each group. The city ran the journalists' code on real applicant data and returned only totals, under an arrangement that complied with the GDPR, the EU's data-protection law.

The journalists had cooperative access to several model versions and the city's code. In June 2025 they published their full method, both model files, the bias-evaluation code, and the city's tables on GitHub. The model files are the original and the reweighted one.

Which groups were wrongly flagged more often reversed between the past data and the live pilot.

What each side says

The caseworker process Slimme Check was meant to improve on was itself flawed. Caseworkers were more likely to wrongly flag Dutch nationals and women. More than half of the investigations caseworkers started found no wrongdoing. Fraud is estimated at roughly 3 percent of applications. The city had about 35,000 welfare recipients at the end of 2024.

Some project staff argued for more testing. They said going back to the caseworker process was itself "a decision to go back to" a biased system. Whether the halt was a governance success, or the abandonment of a tool that could be fixed, is a live disagreement.

The Racism and Technology Center, an advocacy group, reads the same events as the city overriding its citizen panel's clear advice. It argues that adjusting the model for responsible AI missed the deeper question: whether such a system should be built at all.

What this case asks

The case file argues that fairness is not a property you install in a model and then own. It is a relationship between a model and the live population it scores. A check before launch can test that relationship only against past data.

Nearly everything here was built before the pilot. The case file argues the missing piece was a standing check of live results, on a schedule, with the power to halt. The pilot's bias evaluation was the city's own, run once after the pilot. Outside investigators measured the reversed bias many months later.

What this network is drawn from

This network follows the pattern the sources describe, modeled on Amsterdam's Slimme Check pilot. It is not a reconstruction of Slimme Check itself. It shows Slimme Check, the pre-pilot assessments and reviews, the Participation Council, the label reviewer, the investigator, the decision officer, and the applicant and investigation records. It also keeps two checks on the map: the council's overridden advice to stop, and a recheck against live applications, which the city did not have.

What the available tools can and cannot address

A failure pathway is a link between two parts of the network where a mistake made by one part can be passed on to the other. Closing a pathway means mistakes stop passing along it. The work along it goes on. A mistake here is, for example, a wrong investigation-worthy label or a wrong finding entered in the record.

This case has a budget of 10 units. Each tool costs the same at every target level. The pressure Monitoring goes stale is on from the start: no one has to act on what monitoring shows.

The work here is screening applications for investigation. Explore (No Targets) sets no targets. Under Service Targets Only, the targets ask that mistakes stop building on one another, and that Slimme Check keep helping staff get the work done. Before any tool is used, the network sits at a tipping point: mistakes are on the edge of building on one another. A self-correcting network is one where they fade out instead. Assign a challenger, Peer sharing rules, or Mark AI-written records each meets the targets on its own. Many pairs of other tools meet them too.

Under Service and Safety Targets and All Governance Targets, the targets can be met within the budget. Both levels ask you to close every failure pathway, and to keep the work from falling behind, among other targets. Before any tool is used, seven failure pathways are open.

One carries Slimme Check's label to the label reviewer. Two carry each case from the reviewer to the investigator, and from the investigator to the decision officer. Two carry the records out, to Slimme Check as its inputs and to the label reviewer. Two carry findings and decisions into the records.

Every combination within the budget that meets these targets has four tools. Escalate checks closes the label pathway. Mark AI-written records closes the two pathways out of the records. Peer sharing rules or Assign a challenger closes the two hand-offs between staff. Gate record entries or Store less data closes the two pathways into the records. Each of these four combinations costs 9 units. The last unit can buy the stronger setting of Peer sharing rules, Assign a challenger, or Mark AI-written records.

All Governance Targets asks more of the service, and the same four combinations still meet it. Pause AI on alarms also closes the label pathway, but it leaves the work falling behind, so it meets the targets at no level. Check with a second model adds the check named Recheck against live applications. But staff then rely more on the labels, catch fewer mistakes, and fall behind. No combination at the two higher levels includes it.

More is not better here. Using every tool, each at its strongest setting, costs 45 units, four and a half times the budget. It closes every pathway, but the work falls behind. So it meets the targets at none of the three levels that set them.