Truace
Subscribe

Accuracy, completeness and hallucinations in large language model generated statements and references for common rhinologic disorders

Large language models (LLMs) are increasingly adopted for medical documentation, clinical decision-making, and scientific writing. Although these applications demonstrated impressive capabilities, concerns persist regarding their accuracy, fabricated references, and ethical implications (1-3). The Massive Multitask Language Understanding (MMLU) test was introduced as a comprehensive benchmark to evaluate how well LLMs perform across a diverse range of tasks (4). This prospective comparative two part study evalua…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1566· cite & verify
Accuracy, completeness and hallucinations in large language model generated statements and references for common rhinologic disorders
Chatbot by James grills. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0

In brief

A prospective comparative two-part study asked international rhinology society members to rate ChatGPT-3.5 summaries of 100 words on epistaxis, allergic rhinitis, odontogenic sinusitis and chronic rhinosinusitis. The summaries covered background, investigations and management.

The second part checked the references produced by newer models for the same prompts. This matters because LLMs are being adopted in clinical and scientific work, yet the supplied text reports no findings, so how accurate or complete the outputs were remains unknown.

Main points

  1. The study had two parts: summary quality and reference quality.
  2. The topics were epistaxis, allergic rhinitis, odontogenic sinusitis and chronic rhinosinusitis.
  3. Reviewers were international rhinology society members, and otolaryngologists used standardized scales.
  4. Reference tests covered CGPT-5 thinking and non-thinking, Claude Sonnet 4.5 non-thinking and Gemini 2.5.

The problem

Concerns persist that LLMs used for medical documentation and clinical decisions may give inaccurate content and fabricated references, prompting a test on four common nose and sinus disorders.

The rundown

The introduction says LLMs are increasingly adopted for medical documentation, clinical decision-making and scientific writing. It says that despite impressive capabilities, worries remain about accuracy, fabricated references and ethics.

The summary task used ChatGPT-3.5 to generate 100-word statements for each of four disorders. Experts assessed accuracy and completeness of those statements.

A separate evaluation examined references from contemporary model versions given the same prompts. Otolaryngologists rated overall reference quality with standardized scales. The text also mentions the MMLU benchmark as a broad test of LLM performance across tasks.

Sources

  1. Peer-reviewedRhinology Journal2026-10-14

The debate