Moderate evidence
More
AI chatbots matched surgical teams' advice up to 93% of the time in a 250-patient study
Study title: Frontier Large Language Models as Adjunctive Perioperative Decision-Support Tools in Elective Gastrointestinal Surgery: A Prospective Longitudinal Comparison With Consultant-Led Multidisciplinary Care
- Evidence type
- Observational study
- Sample
- n = 250
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2990cite or check it
Study title: Frontier Large Language Models as Adjunctive Perioperative Decision-Support Tools in Elective Gastrointestinal Surgery: A Prospective Longitudinal Comparison With Consultant-Led Multidisciplinary Care
A prospective observational study followed 250 adults having elective gastrointestinal surgery on an Enhanced Recovery After Surgery pathway. It gathered 2,398 clinical assessments from preoperative evaluation through discharge. ChatGPT, Claude, Gemini and DeepSeek each reviewed every assessment, producing 9,592 recommendations that were compared with the consultant-led multidisciplinary team's decisions.
Read the study abstract
BackgroundFrontier large language models (LLMs) are emerging as perioperative clinical decision-support tools, but prospective comparisons with consultant-led multidisciplinary teams (MDTs) remain limited. We evaluated the inter-model agreement, concordance with consultant-led multidisciplinary care, safety, and assessment-level inter-model consensus of four frontier LLMs in elective gastrointestinal surgery within an Enhanced Recovery After Surgery (ERAS) pathway.MethodsIn this prospective longitudinal observat…

54Gain side
How the two sides scoreClose call
58Problem side
The gain
In a prospective study of 250 gastrointestinal surgery patients and 9,592 recommendations, four AI chatbots agreed with the consultant-led team's plan 88.8% to 93.2% of the time, with Gemini highest at 93.2%.
The problem
In the same 250-patient surgery study, 799 chatbot recommendations differed from the consultant-led team's plan, and confirmed safety events occurred in 2.6% to 6.0% of evaluations.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
In brief
Agreement with the team was 93.2% for Gemini, 92.9% for Claude, 91.8% for ChatGPT and 88.8% for DeepSeek. Confirmed safety events occurred in 2.6% to 6.0% of evaluations, so a minority of advice was judged unsafe even though agreement was high.
The four models fully agreed with one another on 1,834 assessments (76.5%), and higher agreement among the models went with higher agreement with the team. The authors say the results support further prospective evaluation of supervised use but do not establish clinical correctness, outcome benefit or suitability for autonomous decisions.
Main points
- The study covered 250 adults and 2,398 longitudinal assessments in elective gastrointestinal surgery under an Enhanced Recovery After Surgery pathway.
- ChatGPT, Claude, Gemini and DeepSeek each gave 9,592 recommendations in total, using identical locked prompts in separate conversations without memory.
- Agreement with the consultant-led team: Gemini 93.2% (κ = 0.87), Claude 92.9% (κ = 0.86), ChatGPT 91.8% (κ = 0.84), DeepSeek 88.8% (κ = 0.79).
- All four models agreed in 1,834 assessments (76.5%); majority agreement in 352 (14.7%), split in 154 (6.4%) and full disagreement in 58 (2.4%).
- Of 799 recommendations that differed from the team's plan, 605 met criteria for formal review, and 95 more were sampled, giving 700 for blinded expert adjudication.
- Confirmed safety events ranged from 2.6% to 6.0% of evaluations.
The full story
Researchers assessed patients from preoperative evaluation through discharge. Each of the 2,398 assessments was independently reviewed by four models with locked prompts, no memory retention and no response regeneration, and the consultant-led multidisciplinary team served as the reference standard.
The primary outcome was agreement with the team. Secondary outcomes included Cohen's κ, agreement by clinical domain, consensus among the models, hallucinations, unsafe recommendations and performance over time, analysed with mixed-effects regression, generalized estimating equations and bootstrap checks.
Where models disagreed with the team, blinded experts reviewed 700 recommendations. The authors conclude that the models matched the team closely under standardized prompting and that stronger consensus among models went with higher agreement, while noting this does not show the advice was clinically correct.
What this doesn’t fix
The authors say the findings do not establish clinical correctness, outcome benefit, or suitability for autonomous decision-making, and results come from standardized prompting conditions.
Sources
1 on file- The American Surgeon™doi:10.1177/00031348261507472
Talk it over
Four chatbots agreed with the consultant-led team 88.8% to 93.2% of the time, yet confirmed safety events occurred in 2.6% to 6.0% of evaluations. How would you weigh the high agreement against those events?
AI chatbots matched surgical teams' advice up to 93% of the time in a 250-patient studyhttps://truvace.com/item/frontier-large-language-models-as-adjunctive-perioperative-decision-support-tool
Four chatbots agreed with the consultant-led team 88.8% to 93.2% of the time, yet confirmed safety events occurred in 2.6% to 6.0% of evaluations. How would you weigh the high agreement against those events?
Answer in the threadThe researchers treated the consultant-led team's decisions as the reference standard, and blinded experts reviewed 700 of the 799 recommendations that differed. What does it mean for the results that the team's plan was the yardstick?
Answer in the threadIf you were a surgeon on the team and a chatbot's advice differed from your plan, what would you want to see before acting on it, and who should make the final call?
Answer in the thread
If you have more time
The four models fully agreed on 1,834 assessments (76.5%). How might a hospital use agreement among models as a signal, and what would it need to check?
Answer in the threadThe authors say the study does not establish outcome benefit. What kind of study, following which patients for how long, would show whether the advice changes recovery?
Answer in the thread
ace
What readers say