Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1797
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-18T06:54:56.152267Z
status: published
lens: g_space
sector: health
headline: Concordance of five large language models with infectious diseases physicians on clinical decision-making: a multi-model comparative survey study
dek: Purpose To evaluate the accuracy of five large language models (LLMs) on a set of infectious diseases (ID) case-based multiple-choice questions and to compare their performance with that of practising ID physicians. Methods A cross-sectional, web-based comparative survey was conducted among ID physicians and residents across Turkey. Ten original case scenarios (30 five-option multiple-choice questions; stratified by difficulty) were validated by a five-expert Delphi consensus panel (two rounds) using IDSA guidel…
gain_title: On a validated 30-question infectious diseases case test, four of five language models scored significantly higher than the physician group's 61.6% accuracy, with Grok reaching 90.0%.
problem_title: (none)
trace_subject: (none)
gain_reading: On a validated 30-question infectious diseases case test, four of five language models scored significantly higher than the physician group's 61.6% accuracy, with Grok reaching 90.0%.
gain_evidence: Four of five LLMs showed statistically significant superiority over the physician group | Physician accuracy was 61.6%
problem_reading: (none)
problem_evidence: (none)
quick_read: Researchers gave ten original infectious diseases case scenarios, made up of 30 multiple-choice questions, to five language models and to physicians and residents in Turkey. The questions were validated by a five-expert Delphi panel against IDSA guidelines and EUCAST 2024 breakpoints. After exclusions, 112 of 129 respondents were analysed, and four of the five models outperformed the physicians.

The result suggests these tools can reproduce guideline-standard answers well, especially for regionally endemic scenarios. However, the authors stress that a structured test is not real clinical practice. The study does not show that models should be used unsupervised, and it does not show general superiority in clinical decision-making.
limitation: The authors say the results show accuracy on a structured multiple-choice test, not general clinical decision-making superiority, and do not establish that models should be used unsupervised in real practice.
tag: Evidence-backed gain
key_points: Five models were tested: Grok, ChatGPT-4o, Gemini 2.5, Claude Sonnet, and DeepSeek-V3, using a standardised prompt in a single session. | Physician accuracy averaged 61.6%, or 18.5 of 30 questions, with a kappa of 0.41 against a 20% chance rate. | Seventeen respondents were excluded for self-reported AI use; they scored higher than included participants, 68.8% versus 61.6%. | Grok scored 90.0% with a Cohen's h of 0.69 against the physician group.
rundown: The study was a cross-sectional, web-based survey of infectious diseases physicians and residents across Turkey. Each question had five options, and difficulty was stratified. The reference standard was set by two Delphi rounds among five experts.

Outcomes included accuracy rate, Cohen's kappa computed from observed response marginals, and an exact one-sided binomial test of each model's score against physician accuracy. Cohen's h was used as the effect size for the difference between proportions.

The models did best on guideline-standard and regionally endemic scenarios. The authors say real clinical practice typically involves collaborative, multidisciplinary input rather than unilateral decisions.
sources:
- peer_reviewed | Infection | https://doi.org/10.1007/s15010-026-03006-3 | 2026-10-06
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

988a5e667c8ce884ff9391cd319f31267b07770b54a2dc14791c78e5e017cd90

previous

0000000000000000000000000000000000000000000000000000000000000000