acelarge language model triage performance in 55 pediatric ED scenarios measured by weighted accuracy accounting for over-triage and under-triage
Source article: Skyer: a novel benchmark for evaluating the effectiveness of large language models in emergency department triage
OBJECTIVES: Emergency department (ED) overcrowding causes diagnostic challenges, prolonged wait times, and impairs appropriate triage, often due to human error and fatigue. Large language models can assist ED staff in triage, improving patient care by mitigating these problems. METHODS: We designed an evaluation method (Skyer benchmark) to assess fifteen large language models, including DeepSeek-R1 (70B, 7B), ChatGPT versions (4, 4.5-preview), Gemini iterations (1.5-pro, 2.0-Pro-experimental, 2.5_03-25, 2.5_05-0…
Contested: both sides are scored from claims and sources, not community votes.
"Sarasota - Doctors Hospital" by roger4336 is licensed under CC BY-SA 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/2.0/.
Researchers designed the Skyer benchmark to evaluate fifteen large language models on 55 realistic pediatric emergency department scenarios using a weighting system for over-triage and under-triage plus three repeat runs for consistency. By the publication date of July 11 2026, ChatGPT-4.5-preview and Gemini-2.5_05-06 had shown 77% and 74% accuracy with mean weights of 377.5 and 365 out of 550, compared to 64% and 253.5 for human experts.
The finding matters because ED overcrowding impairs triage through human error and fatigue, and a reliable assistive tool could streamline quality care. What remains uncertain is whether performance observed in these 55 pediatric cases generalizes to broader populations and real-world workflows, given the authors' own statement that limitations prohibit replacement of human experts.
- Evaluation used 55 realistic clinical pediatric scenarios and a weighting system that accounted for the impacts of over-triage and under-triage
- Fifteen models tested including DeepSeek-R1, ChatGPT 4 and 4.5-preview, Gemini 1.5-pro and 2.5 variants, Mistral-7B, Llama-3.3, Gemma, Qwen-2.5, Phi-4-14B
- Consistency assessed by repeating tests across scenarios three times; top models showed 85% and 82% consistency
In 55 realistic pediatric scenarios evaluated by the Skyer benchmark, ChatGPT-4.5-preview and Gemini-2.5_05-06 achieved higher triage accuracy and weighted scores than human triage experts, with acceptable consistency across repeats.
Despite higher scores, the evaluated large language models have identified limitations that prohibit them from replacing human experts for triage in overcrowded emergency departments.
The rundown
By July 11 2026, researchers had built Skyer to move beyond simple accuracy by weighting over-triage and under-triage harms and by testing consistency across three repeats of each case. The test set comprised 55 realistic clinical pediatric scenarios applied to fifteen models spanning DeepSeek-R1, ChatGPT, Gemini, Mistral, Llama, Gemma, Qwen and Phi families.
Results reported statistically significant differences with p-value < 0.05 and extremely large effect sizes Cohen's D = 2.18 and 1.98 for the two leaders versus human experts. The authors concluded Skyer selected best-performing models that showed consistent results and potential to assist staff rather than replace them in overcrowded EDs.
Authors note limitations that prohibit the best-performing models from replacing human experts, and evaluation was confined to 55 pediatric scenarios with consistency measured over only three repeats.
Sources
- Peer-reviewedCanadian Journal of Emergency Medicine2026-07-11
How should this claim be treated?
The debate