PAN Lab example
Insight Bristol / Think Family Database
The database nobody could audit: a shared child-risk profiling system
Bristol's council and police scored children from a shared database. Staff distrusted the scores. Two models were withdrawn, reportedly with no record of why.
See more
The Think Family Database is a database of Bristol agency records, built by Insight Bristol, a joint team of Bristol City Council and Avon and Somerset Police. The team built at least three models on it. They scored children from 1 to 100 for risk of sexual exploitation, criminal exploitation, and becoming NEET: not in education, employment, or training.
How it was built
Insight Bristol was formed in 2015 under the national Troubled Families Programme, now called Supporting Families. The database fuses roughly 30 to 35 datasets from the council, the police, and other public agencies. It covers about 55,000 families, some 170,000 residents in 2021 reporting. The case file reads this as roughly three in five families with children in the city. Around 450 staff could access it.
The data was gathered through 'legal gateways', legal permissions for agencies to share data, rather than the consent of the people in it. Most people in the records did not know they were there.
The Child Sexual Exploitation model was introduced in 2016. It drew partly on anonymised data about roughly 1,000 children known to have experienced sexual abuse. The sources date the Child Criminal Exploitation model to 2016 or to 2019.
What went wrong
In 2021 the Centre for Data Ethics and Innovation, one of the outside reviewers, reviewed the deployment. Researchers and that review warned that several inputs acted in practice as proxies for poverty. They warned this could produce indirect discrimination. They also warned that, with data gathered through legal gateways, 'legality is not the same as legitimacy'.
An independent evaluation in 2023 judged the risk-scoring models the 'weakest element' of the database. Staff concluded the two exploitation models were 'not fit for operational use'. Reports said victims of exploitation scored below people involved in burglary.
How two models ended
The two exploitation models were quietly withdrawn in 2023. Freedom of Information responses indicate that neither the council nor the police kept records explaining why. They also indicate that the models' source code and variable lists could not be located when auditors sought them.
An ethics committee advising the police analytics reportedly did not revisit the analytics after 2017.
What still runs
The NEET model and the database remain in use as of 2026. In 2025 GOV.UK and Bristol City Council published a description of the NEET model, called an Algorithmic Transparency Record. It describes the model's results as advisory indicators only. It says they should 'not replace professional judgement'.
The Children's Commissioner for England spoke about the tools in June 2026. The Commissioner called it 'deeply concerning that tools used by staff were not fully understood, trusted, and may have compromised their professional judgement'.
What the case shows
Most cases in the Lab's case files fail through over-reliance: people accept a machine's output too readily. Bristol failed the other way. Staff distrusted the scores enough to stop citing them, yet the system kept collecting, kept scoring, and kept writing to a database read across agencies.
The case file reads the database as the center of gravity: one fused store that several models scored from, not any one model. Its lesson is that a distrusted tool is not a harmless one. It still watches people, still shapes the records, and still takes up governance attention.
A system its own owners cannot audit cannot be held to account by anyone else.
What the available tools can and cannot address
A failure pathway is a link between two parts of the network, where a mistake made by one part can be passed on to the other. Here a mistake is, for example, a wrong score or a wrong entry in the database. A pathway counts as closed once few enough mistakes pass along it. The work along it may go on.
This case has a budget of 12 units and starts with no pressure on. When it opens, six failure pathways are open. Two of them, agency data fused into the database and the database read by the models, drain the Privacy gauge.
Understand the system costs 3 units, or 4 under Service and Safety Targets and All Governance Targets, and 6 at its stronger setting. While it is in use, four tools cost 1 unit less, or 2 less at its stronger setting. They are Vet connections, Upgrade model, Require sign-off, and Peer sharing rules. No price falls below 1 unit.
With lingering effects, where damage outlasts its cause, on, the Dynamics setting you first see, three tools act weakly unless Understand the system is also in use. They are Vet connections, Store less data, and Peer sharing rules.
Explore (No Targets) sets no targets. There, with lingering effects on, four pairs of tools, each pair costing 4 units, keep the network's mistakes contained, meaning corrected rather than building on each other. One is Escalate checks with Mark AI-written records. With lingering effects off, Vet connections alone does it, for 3 units.
Under Service Targets Only, the mistakes must be contained and the models must be helping the child-safeguarding work they were built for. With lingering effects on, two pairs meet the targets for 4 units. They are Mark AI-written records with Escalate checks, and Mark AI-written records with Peer sharing rules. With lingering effects off, Vet connections alone meets them for 3 units.
Within the budget, 388 different sets of tools meet the targets under Service Targets Only, counting each tool at either setting. That count uses lingering effects on, where the setting starts. With lingering effects off, 409 sets do.
Service and Safety Targets also ask you to close every failure pathway and keep up with the work. That level runs with lingering effects off. Two sets of tools meet it. The cheapest costs 11 of the 12 units: Vet connections, Mark AI-written records, Store less data, and Keep skills sharp at its stronger setting. The other set adds Require sign-off, for 12 units. The first set also works with Mark AI-written records at its stronger setting, for 12 units.
In those sets, Vet connections stops mistakes passing along agency data fused into the database, the database read by the models, and the models' shared database. Mark AI-written records does the same for staff reading the database. Store less data does it for scores written to the database and casework decisions recorded. Keep skills sharp at its stronger setting is what lets staff keep up with the work.
All Governance Targets is not fully addressable with the available tools. It runs with lingering effects on and asks for the models to be clearly helping the work. Every combination that closes every pathway leaves the models helping less than that. This holds even with the budget lifted, so money is not what stops it. Closing the pathways also cuts into what the models contribute to the work.
No tool here adds the check of scores against outcomes. Assign a challenger and Peer sharing rules add the ethics review of score use. Escalate checks, Review on schedule, Require sign-off, Assign a challenger, Pause AI on alarms, Upgrade model, and Keep skills sharp close no open pathway on their own. With lingering effects on, Vet connections alone closes none either.
More is not better here. Every tool at its strongest setting at once costs 38 units, far over the budget. It closes every pathway, but leaves the models helping the work far less than when the case opens.
Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.
What this models
This example runs on the Think-Family-class shared-database risk-profiling system network: 5 components and 12 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.
Evidence base: 3 assumed · 3 published baseline. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.
Show all 6 assumptions
- assumed
This network follows the shared-database profiling pattern documented in the Insight Bristol and Think Family Database case file. It does not rebuild the actual system.
- baseline
This network assumes staff relied on the scores little, because the sources report distrust and non-use. An independent evaluation judged the risk-scoring models the weakest element of the database. Staff concluded the exploitation models were not fit for operational use. Reports said victims of exploitation scored below people involved in burglary. That is the reverse of the over-reliance seen in most cases in the Lab's case files.
- assumed
This network treats the shared database as the center of the deployment. The agencies' own record systems are drawn as a second store whose data is fused into it. Several models score from that one database. So this network assumes their mistakes are linked through the database's gaps, not independent.
- baseline
This network draws two checks the deployment lacked. The first compares the models' scores with what actually happened, which the case file says was never done. The second is independent review of how staff use the scores. An ethics committee reportedly stopped revisiting the analytics after 2017. Freedom of Information responses indicate no record was kept of why the exploitation models were switched off. Two tools here can add the second check. No tool here adds the first.
- baseline
Outside scrutiny came late, and the deployment's own records of the system could not be found. Auditors could not locate the exploitation models' source code or variable lists. This network shows that through the two checks the deployment lacked, not as any measured rate.
- assumed
The sources document concerns about indirect discrimination and about inputs acting as proxies for poverty. The case file records them. This network follows how mistakes pass between the models, the staff, and the records. It does not model who the people scored are, and estimates no difference in harm between groups.
What this example does not show
Show all 2 limitations
- The documented concerns include a risk of indirect discrimination and inputs acting as proxies for poverty. This example follows how mistakes pass between the models, the staff, and the records. It does not model the families or estimate harm to any group. That risk is documented in the case file, outside any diagram like this one.
- The privacy exposure is shown by the Privacy gauge and by which pathways carry personal data. It is not a measured rate of collection or sharing. This example follows the shape of the withdrawn exploitation models. The NEET model and the database remain in use, and the NEET model's results are described as advisory only: they should not replace staff's professional judgement.
Sources and evidence
What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.
Bristol's Think Family Database drew on roughly 30 to 35 fused council, police, and other datasets covering about 55,000 families (some 170,000 residents in 2021 reporting), and its child sexual and criminal exploitation risk models were quietly withdrawn in 2023 as 'not fit for operational use' after an independent evaluation judged the risk-scoring models the weakest element and staff reported victims of exploitation scoring below people involved in burglary; FOI responses indicate no record was kept of why the models were switched off, and auditors could not locate their source code or variable lists.
empirical- Investigative Sean Morrison, The Bristol Cable with Liberty Investigates, Lighthouse Reports and WIRED, Bristol data tools risked wrongly flagging victims and suspects, Children's Commissioner deeply concerned (2026) https://thebristolcable.org/2026/06/bristol-data-tools-risked-wrongly-flagging-victims-and-suspects-childrens-commissioner-deeply-concerned/
- Investigative Mark Wilding and Matt Burgess, Liberty Investigates and WIRED, Police built a sprawling crime-prediction machine. Some results couldn't be trusted (2026) https://libertyinvestigates.org.uk/articles/predictive-policing-avon-somerset-bristol-police-ai-minority-report/
- Investigative Sean Morrison, The Bristol Cable, Surveillance isn't safeguarding: Think Family and the fight for transparency (2026) https://thebristolcable.org/2026/01/think-family-education-data-gathering-fight-for-transparency/
- Government Bristol City Council, Insight Bristol and the Think Family Database (2025) https://www.bristol.gov.uk/residents/social-care-and-health/children-and-families/insight-bristol
- Investigative Jake Hurfurt (Big Brother Watch), The Bristol Cable, How a police and council database is predicting if your child is at risk of harm (2021) https://thebristolcable.org/2021/07/how-a-police-and-council-database-is-predicting-if-your-child-is-at-risk-of-harm/
Reporting and FOI responses on Bristol's Think Family Database indicate the exploitation models' source code and variable lists could not be located when auditors sought them, and that an ethics committee advising the police analytics reportedly did not revisit the analytics after 2017; a 2021 review warned that data gathered through 'legal gateways' meant 'legality is not the same as legitimacy.'
empirical- Investigative Mark Wilding and Matt Burgess, Liberty Investigates and WIRED, Police built a sprawling crime-prediction machine. Some results couldn't be trusted (2026) https://libertyinvestigates.org.uk/articles/predictive-policing-avon-somerset-bristol-police-ai-minority-report/
- Investigative Sean Morrison, The Bristol Cable with Liberty Investigates, Lighthouse Reports and WIRED, Bristol data tools risked wrongly flagging victims and suspects, Children's Commissioner deeply concerned (2026) https://thebristolcable.org/2026/06/bristol-data-tools-risked-wrongly-flagging-victims-and-suspects-childrens-commissioner-deeply-concerned/
- Investigative Sean Morrison, The Bristol Cable, Surveillance isn't safeguarding: Think Family and the fight for transparency (2026) https://thebristolcable.org/2026/01/think-family-education-data-gathering-fight-for-transparency/
Where this connects
Institutional pressures in this domain
- Workload surge — Demand outruns staffing; per-case attention shrinks and review becomes triage.
- Deadline pressure — Statutory or managerial timeliness rules reward fast approval of machine output over slow disagreement.
- Staff turnover — Experienced skepticism leaves; new staff calibrate their trust on the tool itself.
- Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).
- Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.
All of them in context on the Child welfare & family services domain page.
Levers available here and the patterns behind them
- Escalate checks — State-feedback vigilance
- Vet connections — Connection authorization
- Mark AI-written records — Provenance labeling
- Store less data — Data minimization
- Review on schedule — Oversight cadence & retrospectives
- Require sign-off — Conformity assessment gate
- Understand the system — Understand the system
- Assign a challenger — Structured dissent
- Peer sharing rules — Peer-edge governance
- Pause AI on alarms — Deployment circuit-breaker
- Upgrade model — Improve the model
- Keep skills sharp — Deskilling-arrest mandate
Documented case histories
- Insight Bristol / Think Family Database
- Allegheny Family Screening Tool
- Allegheny Hello Baby
- Douglas County Decision Aide
- The score nobody sees: New York City's concealed severe-harm QA algorithm
- The audit that reached the legislature before it reached the tools: Colorado's safety and risk instruments
- Eckerd Rapid Safety Feedback: origin and spread
- Illinois Rapid Safety Feedback
- The vendor's ledger: Family-Match, the eharmony-derived adoption matcher the states kept coming back to
- ProKid (Netherlands)
- Hackney / Xantura Early Help Profiling
- Sistema Alerta Niñez (Chile)
- The map, not the score: place-based risk terrain and the records it concentrates
- The guardrail's blind side: DC's walled-off child-welfare chatbot that began writing into the case record
- US Birth Match
- Oregon Safety at Screening
- Los Angeles County Project AURA
- What Works for Children's Social Care ML pilots
- New Zealand MSD Predictive Risk Modelling
- Gladsaxe model