48/100

Emerging evidence

Cite
More
EmailLinkedInX

Urologists rate chatbot answers on female urethral stricture; Gemini scored highest, 4.8 of 5

Study title: Guideline Concordance, Information Quality, and Readability of Large Language Model Responses to Female Urethral Stricture Questions: A Comparative Cross-sectional Study

Evidence type
Benchmark or lab evaluation
Sample
n = 30
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Guideline Concordance, Information Quality, and Readability of Large Language Model Responses to Female Urethral Stricture Questions: A Comparative Cross-sectional Study

A cross-sectional study run in January 2026 put 30 standardized questions on female urethral stricture disease to ChatGPT 5.0, Gemini 3.0 and Microsoft Copilot (Basic). Three blinded board-certified urologists scored every answer for accuracy on a 5-point scale. Gemini scored highest at 4.80±0.41.

Read the study abstract

Aim Large language models (LLMs) are increasingly employed to acquire medical information. However, their reliability in addressing rare and technically intricate conditions, such as female urethral stricture disease (FUSD), remains uncertain. This study evaluated the guideline concordance, information quality, reliability, and readability of responses generated by three widely accessible LLMs in response to clinical questions related to FUSD. Methods This comparative cross-sectional study was conducted in Janua…

The gain

In a January 2026 test of 30 guideline-based questions on female urethral stricture, three blinded urologists gave Gemini 3.0 a mean accuracy of 4.80 out of 5, the highest of three chatbots.

In brief

The three chatbots showed different strengths rather than one clear winner. Copilot had the best patient-oriented information quality (EQIP 89.60±4.01) and the easiest reading level (FRES 42.23±3.89, FKGL 12.67±1.04), while Gemini led on reliability, with a modified DISCERN score of 19.0±1.7.

The authors conclude that chatbot-generated information on this rare condition should be checked against current clinical guidance before it is used in care or patient education.

Main points

  1. The 30 questions covered five clinical domains and were built from the 2025 European Association of Urology Guidelines for Urethral Stricture Disease, then refined by expert consensus.
  2. Gemini 3.0 had the highest expert-rated accuracy (4.80±0.41) and modified DISCERN score (19.0±1.7).
  3. Microsoft Copilot (Basic) had the highest EQIP score (89.60±4.01), the highest Flesch reading ease (42.23±3.89) and the lowest grade level (12.67±1.04).
  4. Gemini's modified DISCERN scores were significantly higher than both ChatGPT's and Copilot's (both p<0.001); ChatGPT and Copilot did not differ significantly.
  5. Differences between the models were significant across all evaluated metrics.

The full story

All three models received the same questions under identical conditions: ChatGPT 5.0, Gemini 3.0 and Microsoft Copilot (Basic). Three board-certified urologists, blinded to which model wrote each answer, scored accuracy independently and also rated reliability with modified DISCERN and quality with EQIP.

Readability was measured with the Flesch reading ease score and the Flesch-Kincaid grade level. Between-model comparisons used the Friedman test, followed by Wilcoxon signed-rank tests with Bonferroni correction.

The authors describe distinct performance profiles rather than uniform superiority: Gemini for accuracy and reliability, Copilot for patient-oriented quality and readability. They advise verifying chatbot output on female urethral stricture against current guidance.

Sources

1 on file
  1. Medical Bulletin of Hasekidoi:10.4274/haseki.galenos.2026.93585
    Peer-reviewed2026-11-17

Talk it over

Three questions to start on, two more if you have time
Gemini 3.0 scored 4.80 out of 5 for accuracy, while Microsoft Copilot (Basic) led on patient-oriented quality and readability. Which would matter more to you in answers about female urethral stricture, and why?

Start with a question

Urologists rate chatbot answers on female urethral stricture; Gemini scored highest, 4.8 of 5https://truvace.com/item/guideline-concordance-information-quality-and-readability-of-large-language-mode

  1. Gemini 3.0 scored 4.80 out of 5 for accuracy, while Microsoft Copilot (Basic) led on patient-oriented quality and readability. Which would matter more to you in answers about female urethral stricture, and why?

    Answer in the thread
  2. Three blinded urologists scored 30 questions built from the 2025 European Association of Urology Guidelines, all in January 2026. What can that design tell us about how chatbots answer real patients' questions?

    Answer in the thread
  3. What finding about these chatbots, for example from questions written by patients themselves, would change your view of which one is most useful?

    Answer in the thread
If you have more time
  • Copilot's grade level was 12.67 and its reading ease score was 42.23. What would you look up to judge whether that is readable for the patients the answers are meant for?

    Answer in the thread
  • The authors advise checking chatbot output against current guidance. Who in a clinic or hospital should do that checking, and how would it work?

    Answer in the thread

What readers say