60/100

Moderate evidence

Cite
More
EmailLinkedInX

Chatbots got primary diagnosis right in 28.7% of complex ear, nose and throat cases

Study title: Impact of Prompt Formulation and Case Complexity on Large Language Model Performance in Otolaryngology

Evidence type
Observational study
Sample
n = 120
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Impact of Prompt Formulation and Case Complexity on Large Language Model Performance in Otolaryngology

A comparative observational study tested three AI chatbots (ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro) on clinical information from 80 consecutive outpatients at two otolaryngology centers and 40 published case reports. Primary diagnostic accuracy was 54.6% for moderate cases and fell to 28.7% for complex ones.

Read the study abstract

Objective: To assess how prompt formulation and clinical complexity influence the performance of large language models (LLMs) in real otolaryngology cases. Study design: Comparative observational study. Setting: University hospital study. Methods: Clinical information from 80 consecutive outpatients prospectively recruited at two otolaryngology centers and from 40 published case reports was submitted to ChatGPT-5, Claude-Sonnet-4.5, and Gemini-2.5-Pro using two prompt strategies: a physician-oriented prompt with…

Impact of Prompt Formulation and Case Complexity on Large Language Model Performance in Otolaryngology
"Doctor working in her office while using a computer and taking notes" by Shixart1985, CC BY 2.0.

The problem

In a comparative study of 120 ear, nose and throat cases, ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro reached the primary diagnosis in 28.7% of complex cases versus 54.6% of moderate ones, and lay-language patient prompts were less accurate than physician prompts.

In brief

How the question was worded also changed results. Prompts written in lay patient language produced lower accuracy than physician-style prompts with medical terminology for medical history, symptoms and primary diagnosis. Claude-Sonnet-4.5 and Gemini-2.5-Pro scored significantly higher than ChatGPT-5 on differential diagnosis, additional examinations, treatment and total score.

The authors conclude that outputs need to be read in light of both the prompt and the amount of clinical information given, and they call for structured prompts and supervised clinical use before such tools are integrated into practice.

Main points

  1. The study used 80 consecutive outpatients prospectively recruited at two otolaryngology centers plus 40 published case reports.
  2. Three chatbots were tested: ChatGPT-5, Claude-Sonnet-4.5 and Gemini-2.5-Pro.
  3. Each case was submitted with a physician-oriented prompt using medical terminology and a patient-oriented prompt in lay language.
  4. Two board-certified otolaryngologists classified cases as simple, moderate or complex and rated responses with the Artificial Intelligence Performance Instrument.
  5. Claude-Sonnet-4.5 and Gemini-2.5-Pro significantly outperformed ChatGPT-5 on total score (P < .001).
  6. Primary diagnostic accuracy was 54.6% for moderate cases and 28.7% for complex cases (P < .001).
  7. Models did better on simple cases for history consideration, but on moderate-to-complex cases for physical examination, differential diagnoses and additional examinations.

The full story

Clinical information from 80 consecutive outpatients recruited at two otolaryngology centers and from 40 published case reports was given to the three chatbots. Two board-certified otolaryngologists classified cases as simple, moderate or complex and rated responses with the Artificial Intelligence Performance Instrument; agreement on a subset of complex, practitioner-prompt cases was adequate (ICC = 0.855).

Model choice mattered. Claude-Sonnet-4.5 and Gemini-2.5-Pro significantly outperformed ChatGPT-5 across differential diagnosis, additional examinations, treatment sub-scores and total score (P < .001).

Prompt style and case difficulty shaped results in different ways. Patient-style prompts reduced accuracy for history, symptoms and primary diagnosis, and complexity affected sub-tasks differently: simple cases suited history consideration, while moderate-to-complex cases suited physical examination, differential diagnosis and additional examinations.

Sources

1 on file
  1. Peer-reviewed2026-11-16

Talk it over

Three questions to start on, two more if you have time
Chatbots reached the primary diagnosis in 54.6% of moderate cases and 28.7% of complex ones. How would you weigh that drop against the fact that Claude-Sonnet-4.5 and Gemini-2.5-Pro scored higher than ChatGPT-5?

Start with a question

Chatbots got primary diagnosis right in 28.7% of complex ear, nose and throat caseshttps://truvace.com/item/impact-of-prompt-formulation-and-case-complexity-on-large-language-model-perform

  1. Chatbots reached the primary diagnosis in 54.6% of moderate cases and 28.7% of complex ones. How would you weigh that drop against the fact that Claude-Sonnet-4.5 and Gemini-2.5-Pro scored higher than ChatGPT-5?

    Answer in the thread
  2. Two board-certified otolaryngologists graded complexity and rated the responses, and agreement was checked only on a subset of complex cases with physician prompts. How much would you trust the ratings on the other cases, and why?

    Answer in the thread
  3. If you were a patient typing symptoms in everyday language into one of these chatbots, what would you do with an answer, given that lay prompts were less accurate for symptoms and primary diagnosis?

    Answer in the thread
If you have more time
  • Which clinic or hospital you know might use a structured prompt before a chatbot sees a case, and who would supervise the output?

    Answer in the thread
  • What would a study need to compare to show whether chatbot answers help or mislead people who write in lay language, beyond the 80 outpatients and 40 published case reports?

    Answer in the thread

What readers say