Truace
Subscribe

AI chatbots beat two TB doctors on treatment advice in a 17-case study

Study title: Performance comparison of physicians and large language models in tuberculosis triage, differential diagnosis, and management: a vignette-based comparative study

Researchers compared two AI chatbots, ChatGPT-4o and DeepSeek-R1, with two tuberculosis physicians on 17 standardized cases, 7 simulated and 10 real. In management recommendations, the chatbots scored a median recall of 0.50 against 0.20 for the physicians, and an F1 score of 0.50 against 0.30.

Read the study abstract

BACKGROUND Tuberculosis (TB) remains a major global health challenge, particularly in low- and middle-income countries, where effective triage, diagnosis, and management are often limited. Existing decision-support tools focus on imaging and cannot integrate multi-modal clinical information, constraining their utility in complex clinical scenarios. Large Language Models (LLMs) have shown promise in assisting diagnosis and clinical decision-making in other medical fields, but evidence for their application in TB…

Truvace record2 min read1 sourceEvidence: Emerging · 44/100 — Benchmark or lab evaluation · n = 17 · 1 independent sourcePeer-reviewedTRV-2026-2421 · cite or check it
Performance comparison of physicians and large language models in tuberculosis triage, differential diagnosis, and management: a vignette-based comparative study"Medical laboratories and clinics in Kinwat" by Mouryan, CC BY-SA 4.0.

In brief

Across all tasks, precision was similar (median 0.67 versus 0.50). Reviewers rated chatbot answers higher for suitability, information quality, understandability and operability. The authors conclude the models could assist physicians, not replace them, and may help reduce disparities in TB care, though that is their projection rather than a measured result.

The comparison was a vignette exercise with no patient interventions or randomization. Physicians' answers read more easily in the Chinese readability measure, and the two groups did not differ on safety or conciseness.

Main points

  1. The study was run in 2025 under STARD guidelines and used 17 standardized TB cases, 7 simulated and 10 real.
  2. Responses came from ChatGPT-4o, DeepSeek-R1 and two TB physicians; three TB specialists set the reference standards.
  3. Across all tasks, precision was similar for chatbots and physicians (median 0.67 vs 0.50; P = .35).
  4. Across all tasks, chatbots had higher recall (0.53 vs 0.33) and F1 (0.58 vs 0.33) in management recommendation tasks.
  5. Chatbots scored higher on suitability (3.67 vs 3.00) and information quality (3.33 vs 2.67) on 5-point Likert scales.
  6. Understandability (3.67 vs 3.00) and operability (3.67 vs 3.00) also favored the chatbots.
  7. No differences appeared in conciseness (P = .54) or safety (P = .06).
  8. Physicians' responses were more readable (1.88 vs 2.17; P < .001).

The gain

In a study of 17 tuberculosis cases, ChatGPT-4o and DeepSeek-R1 beat two TB physicians on management recommendations, with recall of 0.50 versus 0.20 and F1 of 0.50 versus 0.30.

The full story

The study tested whether large language models can assist TB physicians in triage, differential diagnosis and management recommendations. It was set against the problem of delayed and unequal TB care in low- and middle-income countries, where existing decision-support tools focus on imaging and cannot combine multi-modal clinical information.

Objective performance was measured with precision, recall and F1 scores against a reference standard from three TB specialists. Subjective ratings used 5-point Likert scales for suitability and information quality, plus safety, conciseness, understandability and operability for management tasks. Group differences were tested with Mann-Whitney U tests.

The authors say the chatbots can serve as adjuncts that enhance management recommendations without replacing physicians, and that their use may improve decision efficiency and help reduce disparities in TB care. The design involved no patient interventions or randomization, so it was not registered as a clinical trial.

Sources

  1. Peer-reviewedBMC Infectious Diseases2026-11-06

The debate