Vanderbilt VSAIL suicide-risk alert

VSAIL, short for Vanderbilt Suicide Attempt and Ideation Likelihood, is a machine-learning model Vanderbilt University Medical Center built in house. It estimates a patient's risk of a suicide attempt within 30 days from routine electronic health records. In a trial, clinicians saw an alert at visits where that risk was 2 percent or higher.

What the trial tested

Vanderbilt ran a randomized trial, registered as NCT05312437 and deployed as Vanderbilt Safecourse. It ran from August 2022 to February 2023 in three Vanderbilt neurology clinics. Vanderbilt chose neurology because some neurological conditions carry a higher suicide risk.

VSAIL flagged 596 of 7,732 visits, about 8 percent, at a risk of 2 percent or higher. Each flagged visit was assigned by chance to one of two alert forms. One was a pop-up the clinician had to dismiss to go on. The other was a passive icon in the chart.

The score, the risk cutoff, and the suicide-risk screen offered, a standard set of questions, were the same in both. Only the form of the alert changed.

What it found

With the pop-up, clinicians chose to run a suicide-risk screen at 42 percent of flagged visits, 121 of 289. With the chart icon, they did so at 4 percent, 12 of 307.

The screen is the Columbia Suicide Severity Rating Scale, a standard set of questions a clinician asks the patient. Documented assessments on it rose to 22 percent, against 8 percent the year before.

Screening stayed the clinician's choice. The pop-up could be dismissed with no required action. About 58 percent of pop-up alerts and 96 percent of chart-icon alerts led to no screen.

What the trial could not show

No suicidal thoughts or attempts were documented in either group during 30 days of follow-up. The trial was explicitly not large enough to measure clinical outcomes. So it showed more screening, a process outcome, but could not speak to reduced harm. As the case file puts it, a gain in process is not a gain in outcome.

The tradeoff the researchers named

The researchers named alert fatigue as the central tradeoff. Alert fatigue is the reflex to dismiss a familiar interruption. Clinicians said they preferred the passive icon, which led to far fewer screens.

Most flags are false alarms. In a 2021 study, clinicians would need to screen 271 of the highest-risk patients to find one suicide attempt. They would need to screen 23 to find one patient with suicidal thoughts. This figure is called the number needed to screen.

Who decides

The clinician at the visit decides whether to run the screen. Vanderbilt's institutional review board, the committee that approves research on people, let the trial run without the usual individual consent. The sources do not say whose consent was waived. One reason given was to protect clinicians who might credibly disagree with VSAIL and decline to screen. The trial left them free to decline.

How the model was checked

The case file calls Vanderbilt's governance comparatively rigorous. Governance included review-board approval, embedded ethicists, and consultation with Vanderbilt's Office of Legal Affairs. The trial was registered in advance. A study of the model running silently, with no alerts, was published before any live alerting.

That silent study ran from June 2019 to April 2020, with 115,905 predictions on 77,973 patients. The model's early risk estimates were miscalibrated: the risks it predicted did not match the rates later seen. Vanderbilt recalibrated it by hand. The sources call this monitoring algorithmovigilance.

Area under the curve measures how well a score ranks a real case above a non-case, where 0.5 is a coin flip. In the silent study it was 0.797 for attempts across the medical center. In behavioral-health settings, where suicidality is most concentrated, it was 0.544.

VSAIL is not a device cleared by an independent regulator. A 2023 study applied it to 260,583 Navy primary-care patients at Naval Medical Center Portsmouth. Its area under the curve was 0.77 applied directly, and 0.92 after retraining.

Where to look

Watch the step from VSAIL to the clinician. The case offers a rare clean measurement: hold the score, the cutoff, the population, and the screen constant, and change only how the alert is delivered. Screening moved roughly tenfold.

What matters here is not the model's accuracy. It is how the alert is delivered, and the attention it commands.

The same step has a second edge. Force attention too bluntly with a flag that is mostly false alarms, and clinicians may learn to dismiss the very screen the alert exists to prompt.

What the available tools can and cannot address

A failure pathway is a link between two parts of the network where a mistake made by one part can be passed on to the other. A pathway counts as closed once only a few mistakes pass along it. The work along it goes on.

Contained means the network's mistakes are corrected rather than building on one another. Strained means the work the system supports is not keeping up.

This case has a budget of 10 units. It starts at the tipping point, where mistakes are on the edge of building on one another rather than being corrected. Three failure pathways start open: Record fields read by VSAIL, Screening recorded, and One model scores each patient.

Under Explore (No Targets), which sets no targets, almost any one tool keeps the mistakes contained. Under Service Targets Only, the same single tools also meet the targets. That level asks for the automated system to be helping the work.

The cheapest cost 2 units each: Mark AI-written records, Escalate checks, Keep skills sharp, Assign a challenger, and Peer sharing rules. Review on schedule does it alone only at its stronger setting, for 4 units.

Upgrade model does it alone at its standard setting, for 3 units, with lingering effects on, the setting you first see. With them off, it needs its stronger setting, for 5 units.

Understand the system costs 3 units under Explore and Service Targets Only, and 4 at the two higher levels. While it is on, four tools cost 1 unit less: Review the riskiest first, Assign a challenger, Peer sharing rules, and Upgrade model. At its stronger setting, Deep research, for 6 units, they cost 2 less, never below 1.

Under Service and Safety Targets and All Governance Targets, this case is not fully addressable with the available tools. Both levels ask you to close every failure pathway, keep the work from being strained, and have the system helping. All Governance Targets asks for it to be clearly helping.

Closing all three pathways fits the budget. The cheapest ways cost 7 of the 10 units: Mark AI-written records, Peer sharing rules, and either Store less data or Gate record entries. Every combination that closes all three includes those tools.

Within the budget, every such combination leaves the work strained, or leaves the automated system not helping the work. Adding a fourth tool that relieves the strain leaves the system not helping.

More money does not change that. With the budget set aside and every tool at its stronger setting, every failure pathway closes, and the automated system ends up hurting the work.

That is a finding about the deployment, not a gap in your approach.