Truace
Subscribe

Gemini matched experts on 92% of prostate cancer questions, Claude on just 56%

Study title: Evaluation of large language models for clinical decision support in prostate cancer: a multidisciplinary Delphi study

Eleven experts rated how well three AI chatbots answered 25 structured questions drawn from the 2025 European Association of Urology prostate cancer guidelines. After two Delphi rounds, consensus agreement was reached for 92% of Gemini responses, 80% of ChatGPT responses and 56% of Claude responses.

Read the study abstract

Aim: Large language models (LLMs) may help interpret clinical guideline content. However, their performance in guideline-based prostate cancer (PC) scenarios remains insufficiently validated. This study evaluated the agreement between a multidisciplinary expert panel and responses generated by three leading LLMs (Claude, Gemini and ChatGPT) when addressing structured, guideline-derived clinical questions on PC. Materials & methods: A multidisciplinary tumor board created 25 structured questions based on the 2025…

By the Truvace desk2 min read1 sourceEvidence: Emerging · 44/100 — Benchmark or lab evaluation · n = 25 · 1 independent sourcePeer-reviewedTRV-2026-2368· cite & verify
Evaluation of large language models for clinical decision support in prostate cancer: a multidisciplinary Delphi study
Role of Pharmaceutical Personnel In Tumor Board Closing the Gap of Cancer Care by Kauke Zimbwe. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0
Trace impact readingContested
P 45The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 45The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Strong agreement, defined as a median score of at least 7 out of 9, was reached for 24 of 25 Gemini answers (96%), 22 of 25 ChatGPT answers (88%) and 14 of 25 Claude answers (56%). Claude also drew explicit disagreement on three questions (12%).

The gaps matter because the models performed differently on the same guideline material, and the hardest questions involved complex cases, advanced imaging, genetic testing, limited data or controversial topics. The authors stress expert oversight, continuous validation and guideline formats that both clinicians and AI can interpret.

Main points

  1. A multidisciplinary tumor board wrote 25 structured questions based on the 2025 European Association of Urology prostate cancer guidelines.
  2. Guideline figures and tables were converted into plain-language summaries before being given to the models.
  3. Each of Claude, Gemini and ChatGPT received a simple, unmodified query for every question.
  4. Eleven experts rated agreement independently in a two-round modified Delphi process with predefined consensus criteria.
  5. Consensus agreement: Gemini 23/25 (92%), ChatGPT 20/25 (80%), Claude 14/25 (56%), p = 0.0033.
  6. Median scores of at least 7 out of 9: Gemini 24/25 (96%), ChatGPT 22/25 (88%), Claude 14/25 (56%), p = 0.0002.
  7. Most consensus difficulties arose in complex cases, advanced imaging, genetic testing, and questions with limited data or controversial topics.

The gain

Gemini's answers reached expert consensus on 23 of 25 guideline-based prostate cancer questions (92%) and ChatGPT's on 20 of 25 (80%), when given plain-language versions of the guidelines.

The problem

Claude reached expert consensus on only 14 of 25 prostate cancer questions (56%), drew explicit disagreement on three, and all models struggled with complex cases, advanced imaging, genetic testing and controversial topics.

The rundown

The study tested whether general-purpose chatbots can interpret clinical guidelines for prostate cancer. A tumor board built 25 questions from the 2025 European Association of Urology guidelines, and the guideline figures and tables were rewritten as plain-language summaries so the models could use them.

Each model, Claude, Gemini and ChatGPT, received a simple, unmodified prompt. Eleven prostate cancer management experts independently rated agreement with each response using a two-round modified Delphi method with predefined consensus thresholds.

Gemini performed best, with consensus on 92% of responses, versus 80% for ChatGPT and 56% for Claude. Claude's answers drew explicit disagreement on three questions after the second round, and the differences between models were statistically significant.

The authors conclude that advanced LLMs given plain-language guideline adaptations can align strongly with expert consensus in routine situations, but performance varies and is weaker in complex cases. They call for expert oversight, continuous validation and standardized guideline formats.

What this doesn’t fix

Consensus difficulties clustered in complex cases, advanced imaging, genetic testing and questions with limited data or controversial topics, and performance varied by model, so the findings support only routine situations with expert oversight.

Sources

  1. Peer-reviewedJournal of Comparative Effectiveness Research2026-11-04

The debate