PAN Lab example
Chicago Public Schools On-Track indicator
The rule a teacher can explain
Chicago Public Schools uses a readable rule, not AI, to flag ninth graders as off track to graduate. Graduation rates later rose to record highs.
See more
The On-Track indicator is a simple rule, not AI, and it learns nothing from data. Chicago Public Schools turned it into district practice from research by the University of Chicago Consortium on School Research. It flags a ninth grader who has not earned enough course credits, or who has failed more than one core course.
How it is used
Teachers and counselors see which ninth graders the rule flags as off track, while the year can still be changed. The response is theirs: a conversation, a schedule change, or a credit-recovery plan. The district gave the flag to staff in each school to act on, not a score nobody could explain.
In the years after the district committed to the practice, its graduation rate rose to record highs.
Where the rule came from
The consortium studied the district's own students. It found that the simple ninth-grade condition predicts high school graduation with about 85 percent accuracy. It is not a model and not a score. It is a rule, computed from a transcript and readable by anyone.
The consortium's central insight was that ninth-grade course performance, a condition schools can act on, predicts graduation better than the fixed characteristics students arrive with. The rule names a condition to change, not a fixed label.
Why it can be explained
The rule cannot be opaque. A teacher can tell a student exactly why the flag is on: you are short of credits, or you have two core failures. The teacher can also say exactly what would turn it off.
So the student the flag is about can contest it, correct it, and act on it. No disclosure policy or appeal process has to be built, because the explanation is the flag. Its logic can be contested by a fifteen-year-old.
It has no learned bias to audit, because it learned nothing. It reads course outcomes recorded for purposes far older than itself.
What the sources say about the limits
The accuracy figure and the rise in graduation are associational. No randomized trial assigned schools to use the indicator. A district that adopts it is usually doing other things too.
The benefit runs through the response by staff, not through the flag. A list of flagged students helps nobody unless staff have time to act on it. The district's practice resourced the attention, and the case file says that resourcing, not the arithmetic, is where the benefit lives.
How the case file reads it
The case file calls this the counter-case among the Lab's education examples. It reads it against another case in this domain, Wisconsin's statewide Dropout Early Warning System. That system combined several models into one score to label every grade 6 to 9 student's risk of not graduating. The state stopped publishing its dashboards in 2023.
The two took on the same task: flagging ninth graders at risk. This one used a readable rule and ended in record graduation rates. The case file draws the lesson that opacity was never the price of prediction here.
In the case file's reading, a rule explainable in one sentence outperformed in practice a model whose own state agency could not say what it had learned. The deployment around the rule could do things the deployment around the model could not.
The best-documented early-warning success in this domain used the least technology. It is a transparent rule, pointed at a changeable condition, and wired to resourced human attention. Its accuracy is published, and its results are measured at district scale.
What the available tools can and cannot address
A failure pathway is a link between two parts of the network where a mistake made by one part can be passed on to the other. A pathway is closed when mistakes stop passing along it. Each tool has a standard setting and a stronger one. The service is the useful work the flag does for staff: showing them which ninth graders the rule flags while the year can still be changed.
Explore (No Targets) sets no targets. Under Service Targets Only, the targets can be met in many ways. Counting the stronger settings, 62 different sets of tools meet them within this case's 9 budget units. Every tool is in at least one of them.
The cheapest is Mark AI-written records alone, for 2 units. In the Lab, on this network, it stops mistakes passing along the three pathways that read the student transcript: to the rule, to staff, and to the consortium. The sources describe no machine-written entries in the transcript, and no such marking in the district. So the Lab credits this tool with an effect the deployment gives it nothing to act on. Understand the system at its stronger setting also meets the targets alone, for 6 units. Escalate checks with Understand the system costs 4 units, because Understand the system, at its standard setting, makes Escalate checks 1 unit cheaper.
More tools are not better here. Every tool at once, at its stronger setting and ignoring the budget, misses the Service Targets Only targets. The staff time and delay the tools add cancel what the flag adds to the work.
Under Service and Safety Targets and under All Governance Targets, this case is not fully addressable with the available tools. Both levels ask you to close every failure pathway. Two stay open whatever you choose, even with every tool at once.
They are staff checking the flag against the transcript, and teachers recording grades and credits. None of the tools offered for this case acts on either. The tools offered are ones the district holds, such as training, the resourcing of staff attention, and the schedule of reviews. That is a finding about the deployment, not a flaw in your choices.
Open this example in PAN Lab v0.1 to apply pressures and levers and watch what the system does.
What this models
This example runs on the On-track-indicator-class the student can read network: 4 components and 8 pathways between them. Every context in the Lab is a stylized model, never a reconstruction of any actual deployment, and each assumption behind it carries a provenance label.
Evidence base: 3 assumed · 1 published baseline. In the Lab, the shaded evidence band behind each headline readout draws its width from the least-established class below.
Show all 4 assumptions
- assumed
This example draws the checks on the rule as running from the start. The consortium published the rule's accuracy, about 85 percent, and keeps studying the district's outcomes. The district turned that research into practice, so this example draws the link from research to practice as running. The research sources behind this case say the rule writes almost nothing back and learns nothing. So this example draws no pathway from the rule to the transcript, and none from the rule to itself. It assumes a heavy workload: every ninth grader in a very large district. It also assumes enough staff capacity to meet that workload. The sources credit the benefit to that resourced attention in each school.
- assumed
The rule can be contested because anyone can read it. This example draws that on the pathway where staff check a flag against the transcript. When the flag is a readable condition, the student it is about can check it, dispute it, and change it. The explanation is the flag. The case file says this domain's opaque deployments cut that correction loop. Here it is open by design. The rule also points at something schools can change: ninth-grade course performance. The consortium found that it predicts graduation better than the fixed characteristics students arrive with. The rule names a condition to change, not a fixed label.
- baseline
The accuracy figure and the rise in graduation are associational, at district scale. No randomized trial assigns schools to use the indicator. A district that adopts it is usually doing other things too. The benefit runs through the resourced response by staff, not through the flag. A list of flagged students helps nobody unless staff have time to act on it. Nothing here computes or claims an effect of the indicator alone.
- assumed
This example computes no student outcome. It shows how mistakes can pass between the district's rule, its staff, its transcripts, and the consortium's study. Students are outside it. The published accuracy, the graduation record, and the staff response to flags come from the case file. Nothing in this diagram computes them.
What this example does not show
Show all 2 limitations
- This example shows no student outcome. It draws only the district: its rule, its student transcripts, its school staff, and the consortium's study. Students are outside the network. The published accuracy, the graduation record, and the staff response to flags come from the case file. Nothing here computes them.
- The accuracy figure and the rise in graduation are associational, at district scale. No randomized trial assigned schools to use the indicator. Nothing here claims that the indicator alone caused the rise. The sources place the benefit in the resourced response by staff, not in the flag.
Sources and evidence
What this example rests on, claim by claim. Every entry resolves to the same ledger the Evidence Registry publishes.
A large urban school district operationalized a transparent ninth-grade indicator - course credits earned plus no more than one core-course failure - from consortium research showing it predicts high-school graduation with about 85 percent accuracy, and wired it to school-level attention rather than to an opaque score. District graduation rates subsequently rose to record highs. The indicator is a rule anyone can read: a teacher can explain to a student exactly why they are off-track and exactly what would change it, so the contest-and-correction loop that opaque early-warning deployments sever is open by construction.
empirical- Academic Allensworth, E.M., & Easton, J.Q. (2007). What Matters for Staying On-Track and Graduating in Chicago Public Schools. University of Chicago Consortium on School Research. https://consortium.uchicago.edu/publications/what-matters-staying-track-and-graduating-chicago-public-schools-focus-students
The documented limits are as instructive as the result. The indicator's accuracy and the district's graduation rise are associational at district scale - no randomized trial assigns schools to use it - and the benefit mechanism runs through the intervention, not the flag: an indicator wired to attention still depends on the attention being resourced, and the research base's central finding is that what predicted graduation was a condition schools could act on (freshman-year course performance), not a fixed trait of the student. The rule's power is that it points at something changeable, and the district's practice is what changed it.
empirical- Academic Allensworth, E.M., & Easton, J.Q. (2007). What Matters for Staying On-Track and Graduating in Chicago Public Schools. University of Chicago Consortium on School Research. https://consortium.uchicago.edu/publications/what-matters-staying-track-and-graduating-chicago-public-schools-focus-students
Where this connects
Institutional pressures in this domain
- Reviewer bottleneck — One fixed-capacity checking stage sits between AI output and consequence; everything queues behind it.
- Austerity & recovery incentives — Cost-cutting and overpayment-recovery targets tilt the system toward denial and enforcement errors.
- Compliance over substance — Paper controls (sign-offs, checklists) satisfy audits while the behavior they describe erodes.
- Data & policy drift — The world, the intake process, and the rules change under a system trained on how things used to be — two mechanisms with different remedies: the statistical properties of what the system processes move (concept drift), or the mixture of inputs arriving in deployment differs from the mixture it was trained on (covariate shift).
- Vendor opacity — The deploying institution cannot inspect the model, data, or update pipeline it is accountable for.
All of them in context on the Education AI domain page.
Levers available here and the patterns behind them
- Train the staff — AI literacy & boundary rules
- Review on schedule — Oversight cadence & retrospectives
- Check copied records — Reconcile copied records
- Review the riskiest first — Risk-tiered oversight
- Escalate checks — State-feedback vigilance
- Mark AI-written records — Provenance labeling
- Understand the system — Understand the system