Truace
Subscribe

Most leading AI chatbots answer glaucoma exam questions accurately, but one lags

Study title: Comparative Accuracy and Safety of Large Language Models in Glaucoma Knowledge Assessment

Researchers tested four large language models on glaucoma questions from American Academy of Ophthalmology materials: 129 self-assessment items and 32 study-course items. Each question was put to each model once, in a fresh session, with a standardized prompt.

Read the study abstract

Prcis: Across glaucoma MCQs, GPT-5.2, Gemini 3 Pro, and Claude Opus 4.5 were highly accurate, whereas DeepSeek-V3.2 underperformed; potentially harmful wrong answers were uncommon. Purpose: To compare the accuracy and potential patient-safety risk of four large language models (LLMs) on glaucoma multiple-choice questions (MCQs). Methods: DeepSeek-V3.2, GPT-5.2 (via ChatGPT), Gemini 3 Pro, and Claude Opus 4.5 were tested on two American Academy of Ophthalmology datasets: 129 text-only AAO ONE Network glaucoma sel…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 161 · 1 independent sourcePeer-reviewedTRV-2026-2344· cite & verify
Comparative Accuracy and Safety of Large Language Models in Glaucoma Knowledge Assessment
Montreal Glaucoma Clinic: Combat Main Cause of Blindness by National Film Board. Photos by Chris Lund. Public domain

In brief

Three models scored in the 80s and 90s, while DeepSeek-V3.2 landed in the 60s. The findings suggest strong exam-style knowledge in some models, but multiple-choice performance does not show safe use with real patients, and the authors call for clinician oversight.

Main points

  1. On the 129-question AAO ONE set, accuracy was 69.0% for DeepSeek-V3.2, 93.0% for GPT-5.2, 92.2% for Gemini 3 Pro and 96.1% for Claude Opus 4.5.
  2. On the 32-question BCSC set, accuracy was 65.6%, 93.8%, 84.4% and 87.5% respectively; GPT-5.2 beat DeepSeek-V3.2 after correction.
  3. A post hoc screen found 38 of 161 questions (23.6%) could directly produce potentially harmful wrong answers.
  4. Four potentially harmful wrong answers occurred, with no significant difference across models.

The gain

GPT-5.2, Gemini 3 Pro and Claude Opus 4.5 reached 92.2% to 96.1% accuracy on 129 glaucoma self-assessment questions, with rare harmful errors.

The rundown

The study used text-only questions from two American Academy of Ophthalmology datasets. Accuracy was compared with Cochran's Q and pairwise exact McNemar tests, adjusted using the Holm method.

Differences among GPT-5.2, Gemini 3 Pro and Claude Opus 4.5 were not significant on the larger set. DeepSeek-V3.2 was significantly less accurate than all three.

Potentially harmful wrong answers were coded using a prespecified rubric. Harmful errors were uncommon but present, which the authors say underscores the need for clinician oversight.

Sources

  1. Peer-reviewedJournal of Glaucoma2026-11-02

The debate