Truace
Subscribe

AI chatbots matched kidney cancer experts on 92% to 96% of guideline questions

Study title: Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study

An 11-member multidisciplinary panel rated Claude, Gemini and ChatGPT answers to 25 questions drawn from the 2025 European Association of Urology renal cell carcinoma guidelines. Gemini reached consensus agreement on 24 responses (96%); Claude and ChatGPT each reached it on 23 (92%). No response reached consensus in disagreement.

Read the study abstract

Aim: Large language models (LLMs) may facilitate the interpretation of clinical guideline information, but their performance in renal cell carcinoma (RCC) remains insufficiently characterized. We evaluated agreement between a multidisciplinary expert panel and the responses generated by Claude, Gemini and ChatGPT to RCC guideline-based questions. Materials & methods: A multidisciplinary tumor board developed 25 clinical questions based on the 2025 European Association of Urology RCC guidelines. Guideline figures…

By the Truvace desk2 min read1 sourceEvidence: Emerging · 44/100 — Benchmark or lab evaluation · n = 25 · 1 independent sourcePeer-reviewedTRV-2026-2372· cite & verify
Evaluation of large language models for clinical decision support in renal cell carcinoma: a multidisciplinary Delphi study
Improvised peritoneal dialysis in an 18-month-old child with severe acute malnutrition and acute kidney injury a case report P02 by Francis Fredrick, Gudila Valentine. CC BY 2.5 · https://creativecommons.org/licenses/by/2.5
Trace impact readingNegative state
P 57The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 47The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Using a median score of at least 7 on a 9-point scale, Gemini was rated acceptable on all 25 answers, Claude on 24 (96%) and ChatGPT on 23 (92%). The differences among the models were not statistically significant overall or in pairwise comparisons.

The weaker answers clustered in clinically uncertain or individualized scenarios, and panelists noted occasional factual errors, omitted options and limited nuance. The authors say the results show potential for guideline interpretation but not independent clinical reasoning or real-world decision-making, so expert oversight remains essential.

Main points

  1. A multidisciplinary tumor board wrote 25 clinical questions based on the 2025 European Association of Urology renal cell carcinoma guidelines.
  2. Guideline figures and tables were converted into structured, plain-language text summaries before being given to the models.
  3. Each model produced one response per question using direct prompting.
  4. An 11-member expert panel scored responses in a two-round modified Delphi process on a 9-point scale with predefined consensus criteria.
  5. Consensus in agreement: Gemini 24/25 (96%), Claude 23/25 (92%), ChatGPT 23/25 (92%); none reached consensus in disagreement.
  6. No statistically significant differences were found between the models.
  7. Questions that needed more discussion or failed consensus mainly involved clinically uncertain or individualized scenarios.

The gain

When given structured guideline summaries, Gemini won expert consensus agreement on 24 of 25 kidney cancer questions (96%), while Claude and ChatGPT each won it on 23 of 25 (92%).

The problem

Panelists found occasional factual inaccuracies, omitted options and limited clinical nuance, and responses that failed consensus mostly involved clinically uncertain or individualized scenarios.

The rundown

The study tested whether large language models can interpret guideline content for kidney cancer. A multidisciplinary tumor board built 25 questions from the 2025 European Association of Urology guidelines, and the guideline figures and tables were turned into structured plain-language summaries so the models could use them.

Claude, Gemini and ChatGPT each answered every question once with direct prompting. Eleven experts rated the answers in two modified Delphi rounds on a 9-point scale using predefined consensus criteria.

Gemini led numerically, with consensus agreement on 96% of answers and a median score of at least 7 on all 25, against 92% consensus for both Claude and ChatGPT. Medians of at least 7 were reached for 24 Claude and 23 ChatGPT answers, and the gaps were not statistically significant.

Panelists flagged occasional factual inaccuracies, omitted options and limited clinical nuance, with trouble concentrated in uncertain or individualized cases. The authors conclude that the models align frequently with expert consensus when given structured material, but this does not show independent reasoning or real-world performance.

What this doesn’t fix

The models answered only 25 questions from structured guideline summaries, so the results do not demonstrate independent clinical reasoning or real-world decision-making performance.

Sources

  1. Peer-reviewedJournal of Comparative Effectiveness Research2026-11-04

The debate