44/100

Emerging evidence

Cite
More
EmailLinkedInX

AI chatbots misread vaginal images far more often than specialists, a 25-image study finds

Study title: Utility of Artificial Intelligence in Diagnosing Image-Based Vaginal Pathology

Evidence type
Benchmark or lab evaluation
Sample
n = 25
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Utility of Artificial Intelligence in Diagnosing Image-Based Vaginal Pathology

A study of 25 deidentified clinical images of vaginal conditions compared four publicly available AI models (ChatGPT, Gemini, Claude and Grok) with two board-certified urogynecology and reconstructive pelvic surgery physicians. The experts reached 84% top-3 and 66% top-1 diagnostic accuracy, while the AI models averaged 51.8% and 32.5%.

Read the study abstract

Importance Artificial intelligence (AI) tools are increasingly used by patients for health-related concerns before formal medical evaluation. However, data evaluating the diagnostic performance of these AI models for vaginal pathology are limited. Objectives The objectives of this study were to evaluate the diagnostic accuracy of publicly available AI models for image-based identification of vaginal pathology and compare their performance with expert physician review. Study design This diagnostic‑accuracy study…

The problem

In a diagnostic-accuracy study of 25 clinical images, four public AI chatbots (ChatGPT, Gemini, Claude and Grok) averaged 51.8% top-3 accuracy versus 84% for two urogynecologists, and recommended urgent evaluation in 77% of cases versus 22% for the experts.

In brief

The work matters because patients increasingly turn to AI tools for health concerns before any formal medical evaluation. The AI models also advised urgent evaluation in 77% of evaluations against 22% for the experts, which could send patients to urgent care that is not needed.

Performance varied by model. ChatGPT and Gemini scored highest, each with 68% top-3 accuracy, and with top-1 accuracy of 44% and 52% respectively. Malignancy was a small subset of three cases, so conclusions about cancer detection rest on very few images.

Main points

  1. The study used 25 deidentified clinical images representing various vaginal conditions.
  2. Four public models (ChatGPT, Gemini, Claude, Grok) and two board-certified urogynecology physicians each gave top-3 and top-1 diagnoses using a standardized prompt.
  3. Experts reached 84% top-3 and 66% top-1 accuracy; AI averaged 51.8% top-3 and 32.5% top-1.
  4. ChatGPT and Gemini performed best, with 68% top-3 accuracy each and top-1 accuracy of 44% and 52%.
  5. In the three malignant cases, the correct diagnosis appeared in the top 3 in 83.5% of evaluations across both groups, with AI slightly more likely to rank malignancy first.
  6. AI recommended urgent evaluation in 77% of evaluations versus 22% for experts.

The full story

Researchers presented 25 deidentified clinical images of various vaginal conditions to four publicly accessible AI models and to two board-certified urogynecology and reconstructive pelvic surgery physicians. Each was asked, with a standardized prompt, for top-3 and top-1 diagnoses, whether the image showed malignancy, a confidence rating from 0 to 100% and a triage recommendation.

Experts scored 84% on top-3 accuracy and 66% on top-1. The AI systems averaged 51.8% and 32.5%, with ChatGPT and Gemini the best performers at 68% top-3 accuracy each and 44% and 52% top-1 accuracy.

For the three malignant cases, the correct diagnosis was among the top three in 83.5% of evaluations across both groups, and AI was slightly more likely to rank malignancy first. The AI models recommended urgent evaluation in 77% of evaluations against 22% for the physicians. The authors conclude that patient-facing AI has a limited but evolving role in image-based triage.

Sources

1 on file
  1. Urogynecologydoi:10.1097/spv.0000000000001954
    Peer-reviewed2026-11-13

Talk it over

Three questions to start on, two more if you have time
The four chatbots averaged 51.8% top-3 accuracy against 84% for the two urogynecologists, and advised urgent evaluation in 77% of evaluations against 22%. Which of those two gaps would matter more to a patient, and why?

Start with a question

AI chatbots misread vaginal images far more often than specialists, a 25-image study findshttps://truvace.com/item/utility-of-artificial-intelligence-in-diagnosing-image-based-vaginal-pathology

  1. The four chatbots averaged 51.8% top-3 accuracy against 84% for the two urogynecologists, and advised urgent evaluation in 77% of evaluations against 22%. Which of those two gaps would matter more to a patient, and why?

    Answer in the thread
  2. Only 3 of the 25 images showed malignancy, and two physicians set the expert benchmark. How should that small, narrow sample shape what anyone concludes about cancer detection by these chatbots?

    Answer in the thread
  3. If you were a patient who had asked ChatGPT or Gemini about a concern and was told to seek urgent evaluation, what would you do next, and what would help you decide?

    Answer in the thread
If you have more time
  • Which kinds of vaginal conditions or patients might be missing from 25 deidentified images, and how could that change the chatbots' scores?

    Answer in the thread
  • The authors describe a limited but evolving role for patient-facing AI in image-based triage. Who should decide where those limits sit, and what evidence should they ask for?

    Answer in the thread

What readers say