aceLarge Language Model Performance and Clinical Reasoning Tasks
Importance: Large language models (LLMs) are increasingly marketed for clinical use, yet their ability to replicate full-spectrum clinical reasoning remains uncertain. Existing evaluations often rely on multiple-choice examinations that do not reflect the complexity of patient care. Objectives: To evaluate the longitudinal clinical reasoning ability of state-of-the-art LLMs and to introduce a multidimensional, clinically meaningful benchmark for clinical-grade artificial intelligence (AI). Design, Setting, and P…
"Doctors Hospital Renovation" by Steve Snodgrass is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.
Researchers evaluated 21 off-the-shelf large language models, including GPT-5, Claude 4.5 Opus, Gemini 3.0 and Grok 4, on 29 standardized MSD Manual clinical vignettes representing 16,254 responses scored by medical students. Using the PrIME-LLM composite across differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous reasoning, scores ranged from 0.64 to 0.78.
The pattern matters because high final-diagnosis accuracy coexisted with systematic failure to generate appropriate differentials, with failure rates above 0.80 for differential diagnosis in all models. That gap, obscured by traditional multiple-choice benchmarks, suggests current off-the-shelf models remain unsafe for autonomous clinical use and need evaluation that captures uncertainty navigation.
- Study tested 21 off-the-shelf LLMs including GPT-5, Claude 4.5 Opus, Gemini 3.0 Flash and Pro, and Grok 4 on 29 MSD Manual vignettes totaling 16,254 responses.
- Primary outcome was PrIME-LLM score defined as normalized polygonal area across 5 domains: differential diagnosis, diagnostic testing, final diagnosis, management, and miscellaneous reasoning.
- PrIME-LLM scores ranged from 0.64 for Gemini 1.5 Flash to 0.78 for Grok 4, with reasoning-optimized models outperforming nonreasoning models.
Across 29 standardized clinical vignettes, all 21 tested LLMs failed differential diagnosis in over 80% of cases, indicating they have not achieved the reasoning needed for safe clinical deployment.
The rundown
The evaluation used the January 2025 MSD Manual vignettes and introduced the Proportional Index of Medical Evaluation for LLMs (PrIME-LLM) as the normalized polygonal area across five reasoning domains. Analyses from January to December 2025 included ANOVA, t tests, and regression models.
Results showed differential diagnosis was less accurate than diagnostic testing, while final diagnosis, management, and miscellaneous reasoning were more accurate, and most models showed improved accuracy with image inputs. The authors concluded the PrIME-LLM framework provided greater separation than raw accuracy.
Sources
- Peer-reviewedJAMA Network Open2026-04-13
How should this claim be treated?
The debate