Truace
Subscribe

Large language model performance in respiratory medicine and allergy

Source article: Clinical performance and practical limitations of large language models in respiratory medicine

This Journal club discusses three recent studies evaluating large language models (LLMs) in respiratory medicine and allergy. In COPD, ChatGPT-3.5 was rated higher than Bing Chat by respiratory specialists for accuracy, completeness, clarity and safety, although clinically important omissions remained. In asthma, 13 out of 14 models exceeded 85% accuracy on multiple-choice questions and five exceeded 95%, but performance was lower and more variable in objective structured clinical examinations. In allergen immun…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1397· cite & verify
Clinical performance and practical limitations of large language models in respiratory medicine
Conversation with a Sam Harris chatbot (51169748740) by Steve Jurvetson from Los Altos, USA. Public domain
Trace impact readingContested
P 71The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 68The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

A journal club reviewed three studies that tested large language models on respiratory and allergy tasks. Specialists rated ChatGPT-3.5 above Bing Chat for COPD answers. Many models did very well on asthma multiple-choice questions, and ChatGPT-4 got a fair quality score for allergen immunotherapy information.

The pattern matters because it separates where these tools help from where they could mislead. Structured knowledge tasks suit them, but gaps in completeness, transparency and readability remain. The source supports supervised educational use only, and it leaves open how the models would behave with real patients or in autonomous deployment.

Main points

  1. Respiratory specialists rated ChatGPT-3.5 higher than Bing Chat on accuracy, completeness, clarity and safety in COPD, though important omissions persisted.
  2. In asthma, 13 of 14 models exceeded 85% on multiple-choice questions and five exceeded 95%.
  3. Asthma performance was lower and more variable in objective structured clinical examinations.
  4. ChatGPT-4 on allergen immunotherapy had fair overall DISCERN quality but insufficient JAMA benchmark reliability, and its answers were often hard to read.

The gain

In asthma testing, 13 of 14 language models exceeded 85% accuracy on multiple-choice questions, and five exceeded 95%, showing strong performance on structured knowledge tasks.

The problem

Performance dropped and varied in clinical examination formats, and COPD answers had clinically important omissions, so the models are not ready for unsupervised patient counselling or autonomous clinical use.

The rundown

The COPD study compared two chatbots, with respiratory specialists rating ChatGPT-3.5 above Bing Chat across accuracy, completeness, clarity and safety. Even the better-rated tool left out clinically important information.

The asthma work tested 14 models. Most scored very high on multiple-choice questions, but results fell and varied more when the format moved to objective structured clinical examinations, which are closer to clinical application.

For allergen immunotherapy, ChatGPT-4 reached fair overall quality on the DISCERN instrument but did not meet the JAMA benchmark for reliability. Its responses were often difficult to read, which limits usefulness for patient information.

The authors conclude that current evidence supports supervised educational use in respiratory medicine and allergy. It does not support unsupervised patient counselling or autonomous clinical deployment.

What this doesn’t fix

The source is a journal club discussion of three studies, and it says weaknesses persist in completeness, transparency, readability and clinically nuanced application, so findings do not support unsupervised or autonomous clinical use.

Sources

  1. Peer-reviewedBreathe2026-10-13

The debate