Truace
Subscribe

Expert-rated quality and safety of consumer LLM answers to anterior segment ophthalmology questions

Source article: Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions

Objective: To compare expert-rated quality and safety of responses from four contemporary large language models (LLMs)-ChatGPT, Gemini, Claude, and Grok-on expert-level anterior segment ophthalmology questions. Design: Cross-sectional, masked, randomized-display-order, multievaluator comparative study. Participants: Eleven fellowship-trained cornea and anterior segment subspecialists. Methods: Forty-five expert-level questions spanning six anterior segment domains were administered to each LLM under zero-shot co…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-2120· cite & verify
Evaluator-Masked Assessment of 4 Consumer-Facing Large Language Models on Expert Anterior Segment Ophthalmology Questions
Kapitel 3 1 ChatBot internal by Robert Wagner. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0
Trace impact readingContested
P 67The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 66The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Eleven fellowship-trained cornea and anterior segment specialists rated answers from ChatGPT, Gemini, Claude and Grok to 45 expert-level questions. Responses were masked to model identity and shown in random order, and each was scored on accuracy, completeness, clarity, safety and potential for harm.

The results show that the models which looked best on expert-rated quality also showed a heavier tail of severe safety concerns. That pattern could matter for anyone relying on chatbot answers for eye care, but the authors describe it as hypothesis-generating, and it reflects expert preference at one time point rather than verified clinical correctness.

Main points

  1. The study covered 45 questions across six anterior segment domains, each given to four LLMs and rated by 11 subspecialists, for 9900 complete ratings.
  2. Quality differences were significant on all three quality metrics, with Claude and Grok rated above ChatGPT and Gemini.
  3. No statistically significant between-model difference was found in mean safety scores.
  4. Severe-safety flags appeared for 35.6% of questions with Claude and 33.3% with Grok, versus 6.7% for ChatGPT and 11.1% for Gemini, and the pattern persisted when two evaluators had to agree.

The gain

Eleven masked cornea specialists rated Claude and Grok answers to anterior segment questions as higher quality than ChatGPT and Gemini on accuracy, completeness and clarity.

The problem

A larger share of questions drew at least one severe overall-safety rating for Claude and Grok than for ChatGPT and Gemini, even though mean safety scores did not differ significantly.

The rundown

Questions were given under zero-shot conditions in a single prespecified session on January 19, 2026, with memory, browsing and retrieval disabled. No specialist-authored reference answer key was used, so all metrics reflect expert perception.

Evaluators scored five 5-point Likert metrics. The analyses included Friedman tests with Kendall's W, Bonferroni-adjusted pairwise Wilcoxon tests, length-adjusted mixed models and a masking-integrity diagnostic.

The severe-safety comparison was a post hoc analysis (Cochran Q = 23.4, P < 0.001). Inter-rater agreement on individual ratings was low (Krippendorff's α 0.021-0.168), while the 11-evaluator mean showed moderate reliability (ICC(2,k) 0.58-0.69 for quality metrics).

What this doesn’t fix

The findings reflect expert preference rather than validated clinical correctness, were taken at a single time point without a reference answer key, and the severe-safety signal was a post hoc, hypothesis-generating result.

Sources

  1. Peer-reviewedOphthalmology Science2026-09-04

The debate