Truace
Subscribe

Large language models' performance in answering common patient questions about colonoscopy: an expert-based evaluation

Background This study aimed to compare the performance of two large language models (LLMs) the ChatGPT-4 and the Google Bard in delivering medical information about colonoscopy. It also sought to evaluate the models' responses to common patient and caregiver questions in terms of accuracy, applicability, comprehensiveness, and communication quality, as assessed by experts using a rubric-based method. Methods This comparative, expert-based observational study analyzed the responses of both models to 11 colonoscop…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1413· cite & verify
Large language models' performance in answering common patient questions about colonoscopy: an expert-based evaluation
"Endoscopy" by Yuya Tamai is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.

In brief

Forty gastroenterologists from various institutions in Türkiye rated how two chatbots, ChatGPT-4 and Google Bard, answered 11 colonoscopy questions written in Turkish to mimic real patient inquiries. Each question was asked in a separate session with no prior examples, and responses were scored on a rubric covering accuracy, applicability, comprehensiveness, and communication quality.

ChatGPT-4 scored better than Bard in every domain, which suggests that some chatbots can provide patient-facing procedure information that specialists consider reasonably good. The authors still say human oversight is needed for safety and reliability, and results may differ for other model versions, so the findings do not show that the tools are ready for unsupervised use.

Main points

  1. ChatGPT-4 median overall index scores ranged from 4.1 to 4.7 across the four rubric domains.
  2. Google Bard median scores ranged from 3.6 to 4.0, with lower shares of high ratings.
  3. Bard's high-rating shares were lowest for comprehensiveness (55.7%) and applicability (58.4%).
  4. The accuracy difference favoring ChatGPT-4 had p = 0.003 and an effect size of 0.47.

The gain

Gastroenterologists rated ChatGPT-4's Turkish-language answers to common colonoscopy questions highly, with over 70% of ratings at 4 or above in every domain, outperforming Google Bard.

The rundown

The study was a comparative, expert-based observational design. Questions were posed in Turkish using a standardized zero-shot approach, and evaluators were recruited by snowball sampling and submitted ratings online.

The 40 participating gastroenterologists had a mean age of 39.1 years. They scored each response on a rubric across four domains: accuracy, applicability, comprehensiveness, and communication quality.

The authors conclude ChatGPT-4 gave more accurate, comprehensive, and patient-centered responses than Bard, while cautioning that clinical communication still requires human oversight.

Sources

  1. Peer-reviewedBMC Gastroenterology2026-10-01

The debate