PAN Lab example
Hackney / Xantura Early Help Profiling
The pilot that quietly failed: a small council's family profiler
Hackney Council paid Xantura to flag at-risk families, without telling them. Poor data meant the pilot found few new families, so the council dropped it.
See more
The Early Help Profiling System was software built by Xantura, an analytics firm, and run for the London Borough of Hackney until 2019. It matched data from across council services to flag the families it judged most at risk. Each month it sent social workers written summaries on those families, not a numeric score.
What the system did
The system matched records from across the council's services. They included social-care referrals, school attendance and exclusions, youth offending, reports of domestic abuse and antisocial behaviour, truancy, and benefits status. Its aim was to flag families for early help, meaning support offered before problems grow serious.
Inside, it gave each family a risk score. Each month social workers received a short list of the families it judged most at risk, about 20 at a time. The list was written as summaries explaining why each family was flagged. The social worker decided whether to offer support.
What the families were told
The families were not told directly that their data was used this way. Notice came only through a general online privacy notice. The privacy impact assessment recorded no option to opt out. The council argued that disclosing details of the system could prejudice possible interventions.
The council withheld the method as commercially sensitive. It refused requests for the data-sharing agreements. No independent evaluation of the system's accuracy or harm was ever published. So the council could not fully say what it had bought.
How it was bought
By October 2018 Hackney had paid Xantura £361,400 since 2015, perhaps not all for this system. It procured the system through London Ventures, a programme run by London Councils and the accounting firm Ernst & Young. London Councils represents London's 32 borough councils and the City of London. £91,400 came through London Ventures.
How it ended
The council dropped the pilot in 2019. It reported that the available data was more limited and more variable than expected. The system had flagged about 350 families, but it surfaced 7 children the council did not already know about. The council judged that it could not deliver enough new insight to justify further investment.
This was a rare kind of ending. There was no public scandal. The council made a cost and benefit judgment that the system did not work well enough to keep.
Why a small pilot matters
Hackney matters because it is small. Most AI in public institutions is not a statewide system. It is a pilot bought from a vendor by a stretched team. At that scale, buying the system is the main point of control, monitoring is informal, and quiet failure is the likely end.
A pilot that cannot say what data it uses, or how families would ever learn of it, has already failed a governance test. That holds however accurate it is. What finally stopped this one was not the model. It was the records beneath it.
So the useful tools here come before any change to the model. They are purchase terms with teeth, keeping less data, and someone with the authority to ask who approved profiling families without telling them.
What this network is drawn from
This network follows the pattern the case file describes. It is not a reconstruction of the actual system. It shows the profiling system, the council data it matched, the council's privacy and information governance, the social workers, and the family profiles.
What the available tools can and cannot address
A failure pathway is a link between two parts of the network where a mistake made by one part can be passed on to the other. Closing a pathway means mistakes stop passing along it. The work along it goes on.
This case has a budget of 12 units. Each tool costs the same at every target level, except Understand the system, which pays for ongoing study of the deployment. It costs 3 units under Explore (No Targets) and Service Targets Only, and 4 under the two higher levels. While it is on, Store less data, Vet connections, Review on schedule, and Pause AI on alarms each cost 1 unit less. With its stronger setting, Deep research, they cost 2 less, but never less than 1.
Before any tool is used, four failure pathways are open. They are Council data into the profiles, Monthly report to social workers, Family history read by workers, and Records scored for next report. The network is at a tipping point, where mistakes could start building on one another. It is not self-correcting, meaning it does not clear mistakes on its own. The service is strained.
Explore (No Targets) sets no targets. Under Service Targets Only, the targets are not met before any tool is used. That level asks for the network to be self-correcting and for the system to be helping the work. Escalate checks meets them on its own for 2 units, and so does Mark AI-written records. Assign a challenger at its stronger setting meets them alone for 3. With lingering effects off, Vet connections does too, for 3. Lingering effects is a setting in which damage outlasts its cause, on by default. In all, more than 600 different sets of tools within the budget meet them.
Under Service and Safety Targets and All Governance Targets, the targets can be met. Both levels ask you to close every failure pathway and keep the service from being strained, among other targets. The cheapest way is Escalate checks with Mark AI-written records, for 4 units.
Escalate checks has social workers check the monthly reports more closely when monitoring flags trouble. It closes Monthly report to social workers. Every set of tools that meets these two levels' targets includes it. Pause AI on alarms also closes that pathway, but it takes away so much of the system's benefit that no set including it meets the targets.
Mark AI-written records closes the other three pathways. In the Lab it marks machine-written content in the records so readers can weigh it. Here it stands for marking which entries the profiling system wrote, and where each record came from.
Vet connections with Understand the system closes the same three instead. With Escalate checks, that way costs 8 units under these two levels. More than 100 different sets of tools within the budget meet the targets at each of the two levels.
More is not better here. Using every tool on offer, each at its strongest setting, costs 35 units, nearly three times the budget. It meets the targets at none of the three levels that set them, because the system then adds too little to the work.
Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.
What this models
This example runs on the Hackney-class small-authority family-profiling pilot network: 5 components and 12 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.
Evidence base: 3 assumed · 4 published baseline. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.
Show all 7 assumptions
- assumed
This network follows the pattern the case file documents for Hackney's Early Help Profiling System: a small council's family profiling, bought from a vendor. It is not a reconstruction of the actual system.
- baseline
The network assumes social workers treated each monthly report as a lead to consider, and kept full discretion over what to do. The documented failure was limited benefit, not over-reliance. The system surfaced few genuinely new cases.
- baseline
The main exposure assumed here is data use without consent. Council data on families was profiled without telling them directly, and the impact assessment recorded no opt-out. Keeping profiles under codes until a high-risk alert is treated as a protection.
- assumed
The sources document no score saved for each family and later used to retrain the system. The concern that such tools repeat bias in old records is a general expert critique in the sources. It was not measured at Hackney.
- baseline
Two checks appear in the network that did not run at Hackney: an independent evaluation of the system, and scrutiny that would let families know and question it. No evaluation of the system's accuracy or harm was published. The council withheld the method as commercially sensitive and refused requests for the data-sharing agreements. Assign a challenger can start the scrutiny check. No tool on offer starts the independent evaluation.
- baseline
The same system would score every family it profiled. So a flaw in it would repeat across families rather than vary case by case.
- assumed
The network does not show demographics or harm that falls unequally on families. The documented concerns were transparency, consent, and limited benefit. Some evidence hints at a social and economic skew, because a share of the alerts were for families receiving benefits. No independent demographic evaluation was published, and the network estimates no unequal harm to the people served.
What this example does not show
Show all 3 limitations
- The documented harm here concerns transparency and consent: families were profiled without being told directly. It is not a measured difference between groups. Some evidence hints at a social and economic skew, because a share of the alerts were for families receiving benefits. No independent demographic evaluation was published. This example shows how mistakes pass between the parts of a deployment, not who they fall on. It estimates no unequal harm to the people served. Where such harm exists, the case file documents it, and it is measured outside any network like this one.
- Inside, the system gave each family a risk score and checked it against a high-risk cutoff. Social workers were given written summaries, not a number. Nothing here implies that workers saw a numeric score.
- The cost-savings figures linked to tools like this, per council each year and per child, came from the vendor or from promotional material. No independent evaluation produced them. The only published outcome was the council's own finding of limited benefit.
Sources and evidence
What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.
Hackney paid the analytics firm Xantura £361,400 over four years to run an Early Help Profiling System that flagged families for preventive intervention from council data, but scrapped the pilot in 2019 after finding that, despite flagging about 350 families, it surfaced only 7 children previously unknown to the council and the available data was too limited and variable to justify continuing.
empirical- Investigative Hackney Council pays 360k pounds to data firm whose software profiles troubled families, Hackney Citizen (18 October 2018) https://www.hackneycitizen.co.uk/2018/10/18/council-360k-xantura-software-profiles-troubled-families/
- Investigative Town Hall drops pilot programme profiling families without their knowledge, Hackney Citizen (30 October 2019) https://www.hackneycitizen.co.uk/2019/10/30/town-hall-drops-pilot-programme-profiling-families-without-their-knowledge/
Families whose data Hackney's Early Help Profiling System processed were not informed directly: reporting describes families profiled without their knowledge, given notice only through a general online privacy notice, with no option to opt out recorded in the system's impact assessment and the method withheld as commercially sensitive; the council argued that disclosing the system could prejudice potential interventions.
empirical- Investigative Town Hall drops pilot programme profiling families without their knowledge, Hackney Citizen (30 October 2019) https://www.hackneycitizen.co.uk/2019/10/30/town-hall-drops-pilot-programme-profiling-families-without-their-knowledge/
- Academic Redden J., Dencik L. and Warne H., Datafied child welfare services: unpacking politics, economics and power, Policy Studies 41(5), 507-526 (2020), DOI 10.1080/01442872.2020.1724928 https://www.tandfonline.com/doi/full/10.1080/01442872.2020.1724928
- Investigative Hackney Council pays 360k pounds to data firm whose software profiles troubled families, Hackney Citizen (18 October 2018) https://www.hackneycitizen.co.uk/2018/10/18/council-360k-xantura-software-profiles-troubled-families/
A single automated rule set applied uniformly and without human review produced tens of thousands of correlated wrongful fraud determinations in the documented Michigan MiDAS case — one flaw repeating at caseload scale rather than averaging out.
empirical- Government Michigan AG, settlement of civil-rights class action (Bauserman, 2022) https://www.michigan.gov/ag/news/press-releases/2022/10/20/som-settlement-of-civil-rights-class-action-alleging-false-accusations-of-unemployment-fraud
- Investigative IEEE Spectrum, Michigan's MiDAS unemployment system: Algorithm alchemy that created lead, not gold https://spectrum.ieee.org/michigans-midas-unemployment-system-algorithm-alchemy-that-created-lead-not-gold
Where this connects
Institutional pressures in this domain
- Workload surge — Demand outruns staffing; per-case attention shrinks and review becomes triage.
- Deadline pressure — Statutory or managerial timeliness rules reward fast approval of machine output over slow disagreement.
- Staff turnover — Experienced skepticism leaves; new staff calibrate their trust on the tool itself.
- Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).
- Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.
All of them in context on the Child welfare & family services domain page.
Levers available here and the patterns behind them
- Escalate checks — State-feedback vigilance
- Gate vendor updates — Vendor quality gate
- Store less data — Data minimization
- Vet connections — Connection authorization
- Require sign-off — Conformity assessment gate
- Review on schedule — Oversight cadence & retrospectives
- Assign a challenger — Structured dissent
- Understand the system — Understand the system
- Upgrade model — Improve the model
- Mark AI-written records — Provenance labeling
- Pause AI on alarms — Deployment circuit-breaker
Documented case histories
- Hackney / Xantura Early Help Profiling
- Allegheny Family Screening Tool
- Allegheny Hello Baby
- Douglas County Decision Aide
- The score nobody sees: New York City's concealed severe-harm QA algorithm
- The audit that reached the legislature before it reached the tools: Colorado's safety and risk instruments
- Eckerd Rapid Safety Feedback: origin and spread
- Illinois Rapid Safety Feedback
- The vendor's ledger: Family-Match, the eharmony-derived adoption matcher the states kept coming back to
- ProKid (Netherlands)
- Insight Bristol / Think Family Database
- Sistema Alerta Niñez (Chile)
- The map, not the score: place-based risk terrain and the records it concentrates
- The guardrail's blind side: DC's walled-off child-welfare chatbot that began writing into the case record
- US Birth Match
- Oregon Safety at Screening
- Los Angeles County Project AURA
- What Works for Children's Social Care ML pilots
- New Zealand MSD Predictive Risk Modelling
- Gladsaxe model