52/100

Moderate evidence

Cite
More
EmailLinkedInX

AI chatbots misread eye scans: GPT 5.2 Pro got 32.1% of diagnoses right in a 330-scan study

Study title: Comparative evaluation of GPT 5.2 Pro and Gemini 3 Pro large multimodal models for retinal optical coherence tomography interpretation

Evidence type
Benchmark or lab evaluation
Sample
n = 330
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Comparative evaluation of GPT 5.2 Pro and Gemini 3 Pro large multimodal models for retinal optical coherence tomography interpretation

A cross-sectional study tested two commercial multimodal AI models, GPT 5.2 Pro and Gemini 3 Pro, on 330 publicly available retinal optical coherence tomography scans covering 12 diagnostic categories. Gemini 3 Pro got 58.8% of diagnoses right, against 32.1% for GPT 5.2 Pro.

Read the study abstract

Purpose: Large multimodal models (LMMs) capable of interpreting medical images are increasingly accessible to patients, yet their reliability for retinal optical coherence tomography (OCT) interpretation remains uncertain. This study evaluated the performance of two state-of-the-art LMMs in detecting OCT features and generating diagnostic interpretations. Methods: In this cross-sectional study, 330 publicly available retinal OCT B-scans spanning 12 diagnostic categories, including normal scans and common retinal…

The problem

In a cross-sectional study of 330 retinal OCT scans, GPT 5.2 Pro reached 32.1% diagnostic accuracy and Gemini 3 Pro 58.8%, and GPT 5.2 Pro failed to correctly identify any normal scan.

In brief

Both models handled obvious abnormalities better than subtle structural features, and both often labelled normal scans as diseased. GPT 5.2 Pro did not correctly identify a single normal image. The researchers also observed recurring model-specific biases and hallucinated diagnoses.

The authors say these models are increasingly available to patients, so misreadings outside supervised clinical settings are a risk. They call for careful validation and clear usage boundaries before such tools are built into patient-facing workflows.

Main points

  1. The study analysed 330 public retinal OCT B-scans across 12 diagnostic categories, including normal scans and common retinal diseases.
  2. Both models were tested with zero-shot inference, meaning no task-specific training or examples.
  3. Feature detection covered nine predefined OCT features; Gemini 3 Pro scored a macro-averaged F1 of 0.558 versus 0.472 for GPT 5.2 Pro.
  4. Diagnostic accuracy, judged against expert-adjudicated reference standards, was 58.8% for Gemini 3 Pro and 32.1% for GPT 5.2 Pro.
  5. Both models showed limited sensitivity for subtle structural features but did better on visually salient abnormalities.
  6. Normal scans were often misclassified as pathological, and GPT 5.2 Pro identified none of them correctly.
  7. The researchers observed recurrent model-specific diagnostic biases and hallucinated diagnoses.

The full story

The researchers used 330 publicly available retinal OCT B-scans spanning 12 diagnostic categories, including normal scans and common retinal diseases. Two commercially available models, GPT 5.2 Pro and Gemini 3 Pro, were run with zero-shot inference.

The models were assessed on two tasks: detecting nine predefined OCT features, scored with per-feature metrics and macro-averaged F1, and producing free-text diagnostic interpretations, scored against expert-adjudicated reference standards.

Gemini 3 Pro outperformed GPT 5.2 Pro on both tasks, with an F1 of 0.558 versus 0.472 and diagnostic accuracy of 58.8% versus 32.1%. Both did better on visually salient abnormalities than subtle ones. Overcalling of pathology was common, and GPT 5.2 Pro failed to correctly identify any normal images.

The authors conclude that the models can extract clinically relevant information from OCT scans but that diagnostic reliability remains limited. They warn of misinterpretation outside supervised clinical settings and say careful validation and clear usage boundaries are needed.

Sources

1 on file
  1. Peer-reviewed2026-11-16

Talk it over

Three questions to start on, two more if you have time
Gemini 3 Pro got 58.8% of diagnoses right and GPT 5.2 Pro 32.1%, and GPT 5.2 Pro identified no normal scan correctly. How would you weigh the better result against the risk of normal scans being called diseased?

Start with a question

AI chatbots misread eye scans: GPT 5.2 Pro got 32.1% of diagnoses right in a 330-scan studyhttps://truvace.com/item/comparative-evaluation-of-gpt-5-2-pro-and-gemini-3-pro-large-multimodal-models-f

  1. Gemini 3 Pro got 58.8% of diagnoses right and GPT 5.2 Pro 32.1%, and GPT 5.2 Pro identified no normal scan correctly. How would you weigh the better result against the risk of normal scans being called diseased?

    Answer in the thread
  2. The 330 scans were publicly available and the two models ran zero-shot, with no examples or task-specific training. How would that setup differ from the way patients or clinicians would actually use these tools?

    Answer in the thread
  3. Both models did better on obvious abnormalities than on subtle structural features. What result from a new test on subtle scans would convince you these models are ready for patient-facing use, or settle that they are not?

    Answer in the thread
If you have more time
  • Both models did better on visually salient abnormalities than subtle structural features. Which retinal conditions might patients most likely miss if they relied on that pattern?

    Answer in the thread
  • The authors call for clear usage boundaries before patient-facing use. Who in an eye-care service you know should write those boundaries, and what would they say?

    Answer in the thread

What readers say