Moderate evidence
More
AI chatbots beat human protocolers on 428 radiology requests, a single-center study finds
Study title: Guardrail-Based Radiology Protocolling with Large Language Models: Balancing Automation and Oversight
- Evidence type
- Controlled or validated comparison
- Sample
- n = 537
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2896cite or check it
Study title: Guardrail-Based Radiology Protocolling with Large Language Models: Balancing Automation and Oversight
Researchers tested a workflow in which large language models first decide whether a CT or MRI request is safe to automate, and send incomplete or safety-critical ones to a radiologist. Using 537 body imaging requests from one center, they validated six AI pipelines on 428 requests against human protocolers.
Read the study abstract
Safe clinical deployment of large language models (LLMs) for radiology protocolling requires knowing when not to automate. We evaluated a "guardrail-first" workflow that defers incomplete or safety-critical CT and MRI requests to radiologists before assignment. In this single-center retrospective study, 537 body imaging requests were temporally split into prompt development (n = 109) and validation (n = 428) cohorts. An expert radiologist classified requests as protocol-eligible or requiring review. Six LLM pipe…
57Gain side
How the two sides scoreClose call
58Problem side
The gain
In a single-center retrospective study of 428 CT and MRI requests, Gemini 3.1 Pro assigned an exact or equivalent protocol 12.7 percentage points more often than human protocolers (83.6%) and flagged all 38 safety-critical or major-missing-information requests for radiologist review.
The problem
In the same 428-request radiology protocolling study, models sent protocol-eligible requests to radiologists unnecessarily 1.3% to 34.0% of the time, and GPT-5.2 reached only 65.0% clinically appropriate assignment end to end.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
In brief
Gemini 3.1 Pro caught all 38 requests with safety-critical issues or major missing information and reached 96.3% clinically appropriate assignment. Humans achieved 83.6% exact or equivalent assignment, which four of the six pipelines exceeded by 8.6 to 12.7 percentage points.
Performance varied widely. Unnecessary escalation of routine requests ranged from 1.3% to 34.0%, and GPT-5.2 managed 65.0% end-to-end appropriate assignment. The authors say the results support prospective evaluation rather than immediate deployment.
Main points
- The 537 requests were split by time into 109 for prompt development and 428 for validation.
- An expert radiologist judged 57 of 428 requests (13.3%) as needing review: 15 safety-critical, 23 major missing information, 19 minor clarification.
- Six pipelines were tested: Gemini 3.1 Pro, Claude Sonnet 4.6, GPT-5.2, GPT-5.2 high, DeepSeek V3.2 and DeepSeek V3.2 reasoner.
- Sensitivity for the 38 highest-risk requests was 100% for Gemini 3.1 Pro and 94.7% to 97.4% for the others.
- Among 371 protocol-eligible requests, unnecessary escalation was lowest for reasoning-enabled models.
- Clinically appropriate assignment ranged from 65.0% to 96.3%; the top three pipelines did not differ significantly.
The full story
The study was a single-center retrospective analysis of body imaging requests. Requests were split by time, with 109 used to develop prompts and 428 held for validation. An expert radiologist labelled each as protocol-eligible or requiring review.
Each model ran in two stages: review triage, then protocol assignment. The primary outcome was clinically appropriate assignment, meaning an exact or equivalent protocol or a correct escalation to review, with bootstrapped 95% confidence intervals.
Against a human benchmark of 83.6% exact or equivalent assignment, Gemini 3.1 Pro, Claude Sonnet 4.6, GPT-5.2 high and DeepSeek V3.2 reasoner scored higher by 12.7, 12.4, 11.6 and 8.6 percentage points. Sensitivity for the highest-risk requests was 94.7% to 100% across pipelines.
What this doesn’t fix
The work was a single-center retrospective study, and the authors say the findings support further prospective evaluation before clinical use.
Sources
1 on file- Journal of Imaging Informatics in Medicinedoi:10.1007/s10278-026-02496-9
Talk it over
Gemini 3.1 Pro flagged all 38 safety-critical or major-missing-information requests, while the models sent protocol-eligible requests to radiologists unnecessarily 1.3% to 34.0% of the time. How would you weigh catching every risky request against that extra review work?
AI chatbots beat human protocolers on 428 radiology requests, a single-center study findshttps://truvace.com/item/guardrail-based-radiology-protocolling-with-large-language-models-balancing-auto
Gemini 3.1 Pro flagged all 38 safety-critical or major-missing-information requests, while the models sent protocol-eligible requests to radiologists unnecessarily 1.3% to 34.0% of the time. How would you weigh catching every risky request against that extra review work?
Answer in the threadThe 428 requests came from one center and were judged by one expert radiologist. How might that single setting and single labeller shape what the results can tell us about other hospitals?
Answer in the threadIf you were a radiologist receiving the escalated requests, which would you rather see sent to you: more routine requests that did not need review, or fewer with a small chance of missing a risky one? Why?
Answer in the thread
If you have more time
The authors say the results support prospective evaluation rather than immediate deployment. What would a prospective test in a hospital you know need to measure before radiologists trusted it?
Answer in the threadGPT-5.2 reached 65.0% clinically appropriate assignment while Gemini 3.1 Pro reached 96.3%. Who should decide which model, if any, is good enough, and what gap would be too wide?
Answer in the thread
ace
What readers say