Moderate evidence
More
Chatbots got primary diagnosis right in 28.7% of complex ear, nose and throat cases
Study title: Impact of Prompt Formulation and Case Complexity on Large Language Model Performance in Otolaryngology
- Evidence type
- Observational study
- Sample
- n = 120
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2971cite or check it
Study title: Impact of Prompt Formulation and Case Complexity on Large Language Model Performance in Otolaryngology
A comparative observational study tested three AI chatbots (ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro) on clinical information from 80 consecutive outpatients at two otolaryngology centers and 40 published case reports. Primary diagnostic accuracy was 54.6% for moderate cases and fell to 28.7% for complex ones.
Read the study abstract
Objective: To assess how prompt formulation and clinical complexity influence the performance of large language models (LLMs) in real otolaryngology cases. Study design: Comparative observational study. Setting: University hospital study. Methods: Clinical information from 80 consecutive outpatients prospectively recruited at two otolaryngology centers and from 40 published case reports was submitted to ChatGPT-5, Claude-Sonnet-4.5, and Gemini-2.5-Pro using two prompt strategies: a physician-oriented prompt with…

The problem
In a comparative study of 120 ear, nose and throat cases, ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro reached the primary diagnosis in 28.7% of complex cases versus 54.6% of moderate ones, and lay-language patient prompts were less accurate than physician prompts.
In brief
How the question was worded also changed results. Prompts written in lay patient language produced lower accuracy than physician-style prompts with medical terminology for medical history, symptoms and primary diagnosis. Claude-Sonnet-4.5 and Gemini-2.5-Pro scored significantly higher than ChatGPT-5 on differential diagnosis, additional examinations, treatment and total score.
The authors conclude that outputs need to be read in light of both the prompt and the amount of clinical information given, and they call for structured prompts and supervised clinical use before such tools are integrated into practice.
Main points
- The study used 80 consecutive outpatients prospectively recruited at two otolaryngology centers plus 40 published case reports.
- Three chatbots were tested: ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro.
- Each case was submitted with a physician-oriented prompt using medical terminology and a patient-oriented prompt in lay language.
- Two board-certified otolaryngologists classified cases as simple, moderate or complex and rated responses with the Artificial Intelligence Performance Instrument.
- Claude-Sonnet-4.5 and Gemini-2.5-Pro significantly outperformed ChatGPT-5 on total score (P < .001).
- Primary diagnostic accuracy was 54.6% for moderate cases and 28.7% for complex cases (P < .001).
- Models did better on simple cases for history consideration, but on moderate-to-complex cases for physical examination, differential diagnoses and additional examinations.
The full story
Clinical information from 80 consecutive outpatients recruited at two otolaryngology centers and from 40 published case reports was given to the three chatbots. Two board-certified otolaryngologists classified cases as simple, moderate or complex and rated responses with the Artificial Intelligence Performance Instrument; agreement on a subset of complex, practitioner-prompt cases was adequate (ICC = 0.855).
Model choice mattered. Claude-Sonnet-4.5 and Gemini-2.5-Pro significantly outperformed ChatGPT-5 across differential diagnosis, additional examinations, treatment sub-scores and total score (P < .001).
Prompt style and case difficulty shaped results in different ways. Patient-style prompts reduced accuracy for history, symptoms and primary diagnosis, and complexity affected sub-tasks differently: simple cases suited history consideration, while moderate-to-complex cases suited physical examination, differential diagnosis and additional examinations.
Sources
1 on file- Otolaryngology–Head and Neck Surgerydoi:10.1002/ohn.70524
Talk it over
Chatbots reached the primary diagnosis in 54.6% of moderate cases and 28.7% of complex ones. How would you weigh that drop against the fact that Claude-Sonnet-4.5 and Gemini-2.5-Pro scored higher than ChatGPT-5?
Chatbots got primary diagnosis right in 28.7% of complex ear, nose and throat caseshttps://truvace.com/item/impact-of-prompt-formulation-and-case-complexity-on-large-language-model-perform
Chatbots reached the primary diagnosis in 54.6% of moderate cases and 28.7% of complex ones. How would you weigh that drop against the fact that Claude-Sonnet-4.5 and Gemini-2.5-Pro scored higher than ChatGPT-5?
Answer in the threadTwo board-certified otolaryngologists graded complexity and rated the responses, and agreement was checked only on a subset of complex cases with physician prompts. How much would you trust the ratings on the other cases, and why?
Answer in the threadIf you were a patient typing symptoms in everyday language into one of these chatbots, what would you do with an answer, given that lay prompts were less accurate for symptoms and primary diagnosis?
Answer in the thread
If you have more time
Which clinic or hospital you know might use a structured prompt before a chatbot sees a case, and who would supervise the output?
Answer in the threadWhat would a study need to compare to show whether chatbot answers help or mislead people who write in lay language, beyond the 80 outpatients and 40 published case reports?
Answer in the thread
ace
What readers say