Truace
Subscribe

LLM chatbot answers to COPD patient education and self-management questions in Chinese

Source article: Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance

Background: Large language model (LLM) chatbots are increasingly used to provide health information, yet their reliability in addressing chronic obstructive pulmonary disease (COPD) care questions in the Chinese context remains unclear. To evaluate and compare the performance of three publicly available LLM chatbots - ChatGPT-3.5, ChatGPT-4.0 (OpenAI), and ERNIE Bot (Baidu) - in answering chronic obstructive pulmonary disease (COPD) care questions in China. Methods: We performed a cross-sectional comparative eva…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1911· cite & verify
Evaluating COPD care assistance: a comparative study of ChatGPT-3.5, ChatGPT-4.0, and ERNIE bot performance
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.
Trace impact readingContested
P 66The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 68The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers tested ChatGPT-3.5, ChatGPT-4.0 and Baidu's ERNIE Bot on 40 COPD care questions asked in Chinese. Three consultant respiratory physicians rated the answers. Most were judged accurate, and ERNIE Bot scored higher on accuracy than both ChatGPT versions in this Chinese-language setting.

The results suggest chatbots could assist with basic patient education, but accuracy alone does not make advice useful. Answers were thin and gave few concrete steps patients could follow. The authors call for safeguards and clinician oversight, and the study cannot show how patients would actually use or be affected by these answers.

Main points

  1. The study used 40 COPD care questions across 7 prespecified domains tied to patient education and self-management.
  2. Each question was submitted in Chinese in a new chat session to limit context carryover.
  3. Three consultant-level respiratory physicians independently rated accuracy on a 1 to 3 scale.
  4. ERNIE Bot had higher total accuracy scores and a higher share of good ratings than both ChatGPT versions.
  5. Comprehensiveness for good responses averaged about 1.4 to 2.0 out of 5, and PEMAT actionability was limited.

The gain

Most chatbot answers to Chinese-language COPD care questions were rated accurate by respiratory physicians, suggesting a role in basic patient education.

The problem

Responses from the same chatbots lacked comprehensiveness and concrete, patient-executable steps, so they are unsuitable as standalone clinical advice.

The rundown

This was a cross-sectional comparative evaluation of three publicly available chatbots: ChatGPT-3.5 and ChatGPT-4.0 from OpenAI and ERNIE Bot from Baidu. It was motivated by uncertainty over how reliable chatbots are on COPD care questions in the Chinese context.

Accuracy was summed across three raters on a 3 to 9 scale, with a majority-vote category derived. Responses rated good were also scored for comprehensiveness on a five-point scale, and information quality was assessed with a modified DISCERN framework and the PEMAT understandability and actionability domains.

The authors conclude the tools should not serve as standalone clinical advice and need appropriate safeguards and clinician oversight.

What this doesn’t fix

The evaluation was a cross-sectional test of 40 questions in Chinese, and the ERNIE Bot advantage was described only for this Chinese-language evaluation, so results may not generalise to other languages or real patient use.

Sources

  1. Peer-reviewedJournal of Global Health2026-10-19

The debate