Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2368
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-04T18:01:45.838175Z
status: published
lens: trace
sector: health
headline: Evaluation of large language models for clinical decision support in prostate cancer: a multidisciplinary Delphi study
dek: Aim: Large language models (LLMs) may help interpret clinical guideline content. However, their performance in guideline-based prostate cancer (PC) scenarios remains insufficiently validated. This study evaluated the agreement between a multidisciplinary expert panel and responses generated by three leading LLMs (Claude, Gemini and ChatGPT) when addressing structured, guideline-derived clinical questions on PC. Materials & methods: A multidisciplinary tumor board created 25 structured questions based on the 2025…
gain_title: Gemini's answers reached expert consensus on 23 of 25 guideline-based prostate cancer questions (92%) and ChatGPT's on 20 of 25 (80%), when given plain-language versions of the guidelines.
problem_title: Claude reached expert consensus on only 14 of 25 prostate cancer questions (56%), drew explicit disagreement on three, and all models struggled with complex cases, advanced imaging, genetic testing and controversial topics.
trace_subject: LLM answers to guideline-based prostate cancer clinical questions
gain_reading: Gemini's answers reached expert consensus on 23 of 25 guideline-based prostate cancer questions (92%) and ChatGPT's on 20 of 25 (80%), when given plain-language versions of the guidelines.
gain_evidence: consensus in agreement was reached for 23/25 responses (92%) with Gemini, 20/25 (80%) with ChatGPT
problem_reading: Claude reached expert consensus on only 14 of 25 prostate cancer questions (56%), drew explicit disagreement on three, and all models struggled with complex cases, advanced imaging, genetic testing and controversial topics.
problem_evidence: Explicit disagreement was noted in three questions (12%) with Claude after the second round
quick_read: Eleven experts rated how well three AI chatbots answered 25 structured questions drawn from the 2025 European Association of Urology prostate cancer guidelines. After two Delphi rounds, consensus agreement was reached for 92% of Gemini responses, 80% of ChatGPT responses and 56% of Claude responses.

Strong agreement, defined as a median score of at least 7 out of 9, was reached for 24 of 25 Gemini answers (96%), 22 of 25 ChatGPT answers (88%) and 14 of 25 Claude answers (56%). Claude also drew explicit disagreement on three questions (12%).

The gaps matter because the models performed differently on the same guideline material, and the hardest questions involved complex cases, advanced imaging, genetic testing, limited data or controversial topics. The authors stress expert oversight, continuous validation and guideline formats that both clinicians and AI can interpret.
limitation: Consensus difficulties clustered in complex cases, advanced imaging, genetic testing and questions with limited data or controversial topics, and performance varied by model, so the findings support only routine situations with expert oversight.
tag: Dual reading
key_points: A multidisciplinary tumor board wrote 25 structured questions based on the 2025 European Association of Urology prostate cancer guidelines. | Guideline figures and tables were converted into plain-language summaries before being given to the models. | Each of Claude, Gemini and ChatGPT received a simple, unmodified query for every question. | Eleven experts rated agreement independently in a two-round modified Delphi process with predefined consensus criteria. | Consensus agreement: Gemini 23/25 (92%), ChatGPT 20/25 (80%), Claude 14/25 (56%), p = 0.0033. | Median scores of at least 7 out of 9: Gemini 24/25 (96%), ChatGPT 22/25 (88%), Claude 14/25 (56%), p = 0.0002. | Most consensus difficulties arose in complex cases, advanced imaging, genetic testing, and questions with limited data or controversial topics.
rundown: The study tested whether general-purpose chatbots can interpret clinical guidelines for prostate cancer. A tumor board built 25 questions from the 2025 European Association of Urology guidelines, and the guideline figures and tables were rewritten as plain-language summaries so the models could use them.

Each model, Claude, Gemini and ChatGPT, received a simple, unmodified prompt. Eleven prostate cancer management experts independently rated agreement with each response using a two-round modified Delphi method with predefined consensus thresholds.

Gemini performed best, with consensus on 92% of responses, versus 80% for ChatGPT and 56% for Claude. Claude's answers drew explicit disagreement on three questions after the second round, and the differences between models were statistically significant.

The authors conclude that advanced LLMs given plain-language guideline adaptations can align strongly with expert consensus in routine situations, but performance varies and is weaker in complex cases. They call for expert oversight, continuous validation and standardized guideline formats.
sources:
- peer_reviewed | Journal of Comparative Effectiveness Research | https://doi.org/10.57264/cer-2026-0081 | 2026-11-04
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

f5439070ca93ade096fb496e7ee310dc9fa23a96f39075c451cf13e9adccaf0c

previous

0000000000000000000000000000000000000000000000000000000000000000