aceAssessment of accuracy, quality, and readability of large language models in kidney transplantation practice
Purpose: This study aimed to comparatively evaluate the accuracy, overall quality, and readability of responses generated by three contemporary large language models (LLMs)-Grok AI, ChatGPT-5 Plus, and Gemini 2.5 Flash-to clinical questions commonly encountered in kidney transplantation (KT) practice. Methods: Ten open-ended clinical scenarios covering surgical techniques, complication management, immunosuppressive strategies, and perioperative care were independently assessed by two experienced physicians. LLM-…
In brief
Researchers gave Grok AI, ChatGPT-5 Plus and Gemini 2.5 Flash ten open-ended scenarios drawn from kidney transplantation practice, covering surgical technique, complications, immunosuppression and perioperative care. Two experienced physicians scored the replies for accuracy, comprehensiveness, clarity and clinical relevance, and also applied a global quality score.
The findings suggest current chatbots can handle specialist transplant questions at a level physicians rate highly, which may matter for decision support. The evidence is still narrow, however: only ten scenarios were tested, the replies were text-only and assessed at one time point, and hallucination risk remains, so physician oversight is still required.
Main points
- Grok AI had the highest numerical scores, with a rubric mean of 9.5 and a Global Quality Score of 4.7.
- ChatGPT-5 Plus scored 9.4 on the rubric and 4.6 on quality; Gemini 2.5 Flash scored 9.2 and 4.4.
- Differences between the three models were not statistically significant (p > 0.05).
- Flesch Reading Ease scores of 23.85 to 32.19 indicate professional-level text suited to physicians rather than lay readers.
- Inter-rater reliability was high, with Cronbach's alpha between 0.737 and 0.889.
The gain
Grok AI, ChatGPT-5 Plus and Gemini 2.5 Flash gave highly rated, statistically comparable answers to complex kidney transplantation questions, suggesting a possible adjunct decision-support role under physician oversight.
The rundown
The study used ten open-ended clinical scenarios spanning surgical techniques, complication management, immunosuppressive strategies and perioperative care. Two experienced physicians independently assessed each model's answers using a rubric with equally weighted criteria plus the Global Quality Score.
Readability was measured with the Flesch Reading Ease Score, and agreement between raters was tested with Cronbach's alpha and intraclass correlation coefficients. The reading-ease values fall in the range typically considered difficult, which the authors interpret as appropriate for physicians.
The authors conclude that the models perform comparably and could serve as adjunctive clinical decision-support tools. They attach the condition of appropriate physician oversight and name hallucinations, scenario count and single-time-point text evaluation as key limitations.
Sources
- Peer-reviewedWorld Journal of Urology2026-10-16
The debate