Longer AI prompts did not reliably improve screening of sinus patients for biologic drugs

Study title: Model-Dependent Prompt-Component Ablation for Medical LLM Decision Support: Application to ENT Biologics Screening

Researchers built Medotor, a locally run system that uses language models to screen patients with chronic rhinosinusitis with nasal polyps for biologic therapy. They tested five models on the same 163 retrospective cases under five prompt variants, from a minimal prompt to a full structured prompt with added knowledge retrieval.

Read the study abstract

Large language models (LLMs) show promise for clinical decision support, but privacy constraints and unreliable free-text decisions limit deployment. We present Medotor, a locally deployed system for screening patients with chronic rhinosinusitis with nasal polyps (CRSwNP) for biologic therapy, and evaluate prompt design on 163 retrospective cases (132 eligible and 31 ineligible). Five models were tested on the same patients under five variants: a minimal prompt (P0), the full role/safety/structured nine-point p…

By the Truvace deskNovember 3, 20261 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 163 · 1 independent sourcePeer-reviewedTRV-2026-2340· cite & verify

Model-Dependent Prompt-Component Ablation for Medical LLM Decision Support: Application to ENT Biologics Screening — "Henrico Doctors' Hospital, Skipwith Road entrance" by Packer1028 is marked with CC0 1.0. To view the terms, visit https://creativecommons.org/publicdomain/zero/1.0/deed.en/.

In brief

The results show that prompt design cannot be applied uniformly across models: the same change helped one model and badly hurt another. That matters for hospitals considering locally deployed AI under privacy limits, though the study measured classification and output reliability only, not hallucination reduction or real clinical outcomes.

Main points

The test used 163 retrospective cases, 132 eligible and 31 ineligible for biologic therapy.
Best balanced accuracies ranged from 0.720 for DeepSeek V4 Flash to 0.549 for local Gemma 3n.
Removing the nine-point schema improved Qwen 3.5 compared with the full prompt (Holm-corrected p=0.015).
Adding a fixed MemoRAG-derived knowledge block did not significantly improve any model after correction.

The problem

In screening 163 sinus-disease cases, the best balanced accuracy was only 0.720, and a longer full prompt sharply degraded one local model, producing 84 undecidable outputs.

The rundown

Five models were compared on identical patients under five prompt variants: minimal, full role/safety/structured nine-point prompt, no role/safety block, no nine-point schema, and the full prompt plus a shared knowledge block. Every arm kept all 163 attempts, and undecidable or malformed final decisions counted as errors.

Balanced accuracy was the primary metric, analyzed with patient-level bootstrap intervals, exact paired McNemar tests and Holm correction. The top results were 0.713 for Qwen 3.5 397B and 0.704 for GPT-OSS 120B, while local Qwen 2.5 reached 0.657.

The authors conclude that prompt effects are strongly model-dependent. The full prompt sharply degraded the small local Gemma 3n model, which produced 84 undecidable outputs.