aceChatGPT Health performance in a structured test of triage recommendations
ChatGPT Health was launched in January 2026 as OpenAI's consumer health tool and has reached millions of users. Here we conducted a structured stress test of triage recommendations using 60 clinician-authored vignettes across 21 clinical domains under 16 factorial conditions, yielding 960 total responses. Performance followed an inverted U-shaped pattern, with the most dangerous failures concentrated at clinical extremes-nonurgent presentations (35%) and emergency conditions (48%). Among gold-standard emergencie…
"Doctor's Hospital, Laredo, TX IMG 4149" by Billy Hathorn is licensed under CC BY 3.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/3.0/.
In a structured stress test published February 23, 2026, researchers evaluated ChatGPT Health, OpenAI's consumer health tool launched in January 2026, using 60 clinician-authored vignettes across 21 clinical domains under 16 factorial conditions to generate 960 responses, assessing triage recommendations and contextual sensitivity.
The findings matter because the tool has reached millions of users while showing missed high-risk emergencies and inconsistent crisis safeguards, including undertriage of diabetic ketoacidosis and respiratory failure and unpredictable activation for suicidal ideation, raising safety concerns that the authors say warrant prospective validation before consumer-scale deployment, with uncertainty remaining about real-world performance outside vignettes.
- Structured stress test used 60 clinician-authored vignettes across 21 clinical domains under 16 factorial conditions for 960 total responses.
- ChatGPT Health launched January 2026 and has reached millions of users before prospective validation.
- Undertriage of 52% of gold-standard emergencies included diabetic ketoacidosis and impending respiratory failure directed to 24-48 h evaluation.
- Anchoring bias condition where family or friends minimized symptoms produced OR 11.7 shift toward less urgent care in edge cases.
- Crisis-intervention messages for suicidal ideation activated unpredictably, more frequently when no specific method described than when method described.
In a 960-response vignette test, ChatGPT Health undertriaged 52% of gold-standard emergencies, directing diabetic ketoacidosis and impending respiratory failure to 24-48 h evaluation instead of the emergency department, with failures concentrated at clinical extremes and triage shifting toward less urgent care when by-
The rundown
Researchers tested ChatGPT Health, launched January 2026, using 60 clinician-authored vignettes across 21 domains under 16 factorial conditions to produce 960 responses, measuring triage accuracy and sensitivity to contextual factors.
Performance showed an inverted U-shaped pattern with 35% failure for nonurgent and 48% for emergency presentations; among gold-standard emergencies 52% were undertriaged to 24-48 h care, while stroke and anaphylaxis were correctly triaged, and anchoring bias from family minimization produced an odds ratio of 11.7 toward less urgent recommendations.
Crisis-intervention messaging for suicidal ideation activated unpredictably, occurring more frequently when patients described no specific method than when they did, and analyses found no significant effects for patient race, sex and barriers to care but confidence intervals did not exclude clinically meaningful differences.
Sources
- Peer-reviewedNature Medicine2026-02-23
How should this claim be treated?
The debate