Truace
Subscribe

Large language models in the diagnosis and treatment of immunosuppression-related infections in rheumatic diseases: a comparative, vignette-based study

This study compared the diagnostic accuracy, initial test selection, and initial treatment choices of three current-generation Large Language Models (LLMs) with those of two infectious disease specialists in managing infectious complications in patients with immunosuppressed rheumatic disease. This cross-sectional comparative study utilized 100 standardized Turkish-language clinical vignettes, developed by an expert committee, encompassing ten infection categories: viral, bone/joint, respiratory, central nervous…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1683· cite & verify
Large language models in the diagnosis and treatment of immunosuppression-related infections in rheumatic diseases: a comparative, vignette-based study
"MIHANOVICEVA 3 - rheumatology and physiotherapy" by Miroslav Vajdić is licensed under CC BY-SA 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/2.0/.

In brief

Researchers gave 100 standardized Turkish-language vignettes covering ten infection categories in immunosuppressed rheumatic patients to Gemini 3.1 Pro, ChatGPT 5.5, Claude 4.7 Opus and two blinded infectious disease specialists. Answers on diagnosis, initial tests and initial treatment were scored against a guideline-based reference standard set beforehand.

The results suggest language models could assist clinicians with complex infection cases in patients on immunosuppression, particularly rare or opportunistic ones. The evidence comes from a written exercise with a single correct answer and only two human comparators, so how the models would perform in real patient care remains untested.

Main points

  1. Diagnostic accuracy was 91% and 93% for the two specialists versus 99%, 97% and 100% for the three LLMs.
  2. Complete case management reached 83% and 90% for the specialists and 97-100% for the LLMs.
  3. Of 40 pairwise comparisons, 10 were significant after Bonferroni correction, all between humans and models.
  4. The authors say LLMs should not replace specialists without prospective, real-world validation.

The gain

On 100 standardized vignettes, three LLMs answered more items correctly than two infectious disease specialists in diagnosis, work-up and initial treatment, suggesting a possible decision-support role for rare or opportunistic infections.

The rundown

The study was a cross-sectional comparison using 100 vignettes written by an expert committee. They spanned viral, bone/joint, respiratory, central nervous system, gastrointestinal, genitourinary, skin and soft tissue, rare disease, vaccination and prophylaxis, and adverse drug reaction categories.

The reference standard drew on guidelines from ACR, EULAR, KLİMİK, IDSA, ATS, WHO and SANJO/EBJIS and was fixed before data collection. Participants worked closed-book, and each response was marked correct or incorrect.

The pattern held across all three domains (Cochran Q, all p ≤ 0.003). The authors frame the result as performance on a structured written task, not superiority in clinical care.

Sources

  1. Peer-reviewedRheumatology International2026-10-09

The debate