Doctors rate AI chatbot answers to patient questions on HPV throat infection

Study title: Artificial Intelligence Chatbot Accuracy for Patient Education in HPV Infection

Researchers put eight questions that mimic real patient queries about laryngeal HPV infection and recurrent respiratory papillomatosis to three chatbots: Claude, ChatGPT and Gemini. Two fellowship-trained laryngologists, who did not know which model wrote each answer, scored the responses on a 5-point scale.

Read the study abstract

Objectives: Chatbots are artificial intelligence (AI) tools that use large language models to answer questions and simulate human-like conversations. Chatbots are commonly used by patients for self-education regarding medical issues. The accuracy of information obtained through patient self-education searches using Chatbots for Human Papillomavirus (HPV) infection and recurrent respiratory papillomatosis (RRP) is unknown. This work is intended to analyze aspects of chatbot answers to common patient questions reg…

By the Truvace deskNovember 3, 20261 min read1 sourceEvidence: Emerging · 44/100 — Benchmark or lab evaluation · n = 8 · 1 independent sourcePeer-reviewedTRV-2026-2348· cite & verify

Artificial Intelligence Chatbot Accuracy for Patient Education in HPV Infection — "Hus Findery chatbots" by Antti Törrönen is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.

In brief

Patients often turn to chatbots to learn about medical conditions, yet the quality of the answers for this rare airway condition was unknown. The results suggest quality differs by platform, with Gemini standing out. The study text does not show whether any answers could mislead patients or how people would use them in practice.

Main points

Eight patient-style questions were posed to Claude, ChatGPT and Gemini.
Two laryngologists blindly rated each answer for accuracy, comprehensiveness and similarity to their own advice.
Gemini scored highest overall at 4.7, versus 4 for Claude, which scored lowest.
Only comparisons of Gemini with ChatGPT and with Claude were statistically significant (P < 0.0001).

The gain

Gemini's answers to common patient questions on laryngeal HPV and recurrent respiratory papillomatosis scored 4.7 out of 5 from laryngologist raters, with accuracy the highest-rated aspect across platforms.

The rundown

The study scored each chatbot's answers on three aspects: accuracy, comprehensiveness, and similarity to the response the rating physician would give. Gemini had the highest individual scores on each of these aspects.

Across all three platforms, accuracy received the highest rating at 4.4, though the authors note this difference was not robust. A one-way ANOVA found an overall significant difference between platforms.

A post hoc Tukey test showed that pairwise differences were significant only when Gemini was compared individually with ChatGPT and with Claude. Gemini was the only platform showing a robust difference from the others.