Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1681
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-17T10:41:45.159240Z
status: published
lens: trace
sector: health
headline: Human-in-the-Loop Suicide Risk Detection and Clinical Escalation in a Generative AI Mental Health Chatbot: Prospective Implementation Study
dek: Background Mental illness is the leading cause of disability worldwide, affecting more than one billion individuals, and suicide accounts for 720,000 to 800,000 deaths annually. While AI-native mental health chatbots have the potential to fill a sizable gap between treatment need and workforce capacity, their rapid proliferation has outpaced the development of scalable procedures to identify, detect, and respond to suicidal ideation. Despite AI best practice recommendations to incorporate human oversight for saf…
gain_title: A human-in-the-loop protocol combining universal screening and conversational detection was operationally feasible, starting clinician outreach about 11 minutes after risk alerts in a chatbot trial.
problem_title: Clinicians successfully contacted only 30.9% of alerted participants in Phase 1 and 42.4% in Phase 2, leaving many flagged users without completed human assessment.
trace_subject: Human clinician escalation for suicide risk alerts in a generative AI mental health chatbot
gain_reading: A human-in-the-loop protocol combining universal screening and conversational detection was operationally feasible, starting clinician outreach about 11 minutes after risk alerts in a chatbot trial.
gain_evidence: Outreach was initiated a mean of 11.2 minutes after the alert
problem_reading: Clinicians successfully contacted only 30.9% of alerted participants in Phase 1 and 42.4% in Phase 2, leaving many flagged users without completed human assessment.
problem_evidence: A clinician successfully reached 25/81 participants (30.9%) | Clinicians successfully contacted 73/172 participants (42.4%)
quick_read: Researchers tested a protocol in which adults using an AI-native mental health chatbot were screened for suicidal ideation with the PHQ-9. In the second phase, alerts could also be triggered by suicidal disclosures made during chatbot conversations. Alerted users were routed to a human clinician for assessment, safety planning and referrals.

The study suggests that human oversight of mental health chatbots can be run in practice, with outreach starting within roughly eleven minutes on average. Still, most alerted participants in both phases were not reached, and the study does not show whether the procedures improve clinical outcomes, so the safety value remains unproven.
limitation: Contact and clinical assessment were not completed for a sizable share of alerted participants, and the effect of these procedures on clinical outcomes was not characterized.
tag: Dual reading
key_points: In Phase 1, 81 of 624 screened participants (13.0%) endorsed PHQ-9 item 9. | In Phase 2, 133 of 650 participants (20.5%) screened positive at baseline. | Over 30 days in Phase 2, 284 risk-alert events occurred among 172 participants, including 50 conversational suicidal ideation events. | Mean time from alert to outreach was 11.2 minutes in Phase 1 and 11.5 minutes in Phase 2.
rundown: The study was a single-arm implementation trial in which adults were recruited online. Everyone completed pre-treatment PHQ-9 screening, and the Ask Suicide-Screening Questionnaire guided human-led assessment and risk stratification in both phases.

Phase 1 escalated only those who endorsed suicidal ideation on the questionnaire. Phase 2 added detection of implicit or explicit suicidality disclosed in conversation with the chatbot, which produced additional alerts during the 30-day observation window.

Outreach timing ranged from 1 to 54 minutes in Phase 1 and 1 to 58 minutes in Phase 2. Some participants who responded described the approach as acceptable and helpful.
sources:
- peer_reviewed | JMIR Formative Research | https://doi.org/10.2196/106118 | 2026-10-09
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

38a7154c24047d3858a39154467e5e0e044a20d8dc33dd616ffc306bb63bc6e0

previous

0000000000000000000000000000000000000000000000000000000000000000