Truace
Subscribe

Concordance of five large language models with infectious diseases physicians on clinical decision-making: a multi-model comparative survey study

Purpose To evaluate the accuracy of five large language models (LLMs) on a set of infectious diseases (ID) case-based multiple-choice questions and to compare their performance with that of practising ID physicians. Methods A cross-sectional, web-based comparative survey was conducted among ID physicians and residents across Turkey. Ten original case scenarios (30 five-option multiple-choice questions; stratified by difficulty) were validated by a five-expert Delphi consensus panel (two rounds) using IDSA guidel…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1797· cite & verify
Concordance of five large language models with infectious diseases physicians on clinical decision-making: a multi-model comparative survey study
Doctor Wenlock outside St Thomas's Hospital - geograph.org.uk - 3101668 by PAUL FARMER. CC BY-SA 2.0 · https://creativecommons.org/licenses/by-sa/2.0

In brief

Researchers gave ten original infectious diseases case scenarios, made up of 30 multiple-choice questions, to five language models and to physicians and residents in Turkey. The questions were validated by a five-expert Delphi panel against IDSA guidelines and EUCAST 2024 breakpoints. After exclusions, 112 of 129 respondents were analysed, and four of the five models outperformed the physicians.

The result suggests these tools can reproduce guideline-standard answers well, especially for regionally endemic scenarios. However, the authors stress that a structured test is not real clinical practice. The study does not show that models should be used unsupervised, and it does not show general superiority in clinical decision-making.

Main points

  1. Five models were tested: Grok, ChatGPT-4o, Gemini 2.5, Claude Sonnet, and DeepSeek-V3, using a standardised prompt in a single session.
  2. Physician accuracy averaged 61.6%, or 18.5 of 30 questions, with a kappa of 0.41 against a 20% chance rate.
  3. Seventeen respondents were excluded for self-reported AI use; they scored higher than included participants, 68.8% versus 61.6%.
  4. Grok scored 90.0% with a Cohen's h of 0.69 against the physician group.

The gain

On a validated 30-question infectious diseases case test, four of five language models scored significantly higher than the physician group's 61.6% accuracy, with Grok reaching 90.0%.

The rundown

The study was a cross-sectional, web-based survey of infectious diseases physicians and residents across Turkey. Each question had five options, and difficulty was stratified. The reference standard was set by two Delphi rounds among five experts.

Outcomes included accuracy rate, Cohen's kappa computed from observed response marginals, and an exact one-sided binomial test of each model's score against physician accuracy. Cohen's h was used as the effect size for the difference between proportions.

The models did best on guideline-standard and regionally endemic scenarios. The authors say real clinical practice typically involves collaborative, multidisciplinary input rather than unilateral decisions.

Sources

  1. Peer-reviewedInfection2026-10-06

The debate