Truace
Subscribe

Medical AI tool OpenEvidence scored lower than general chatbots on eye surgery exam questions

Study title: OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an Advantage?

Researchers gave 225 multiple-choice questions from the refractive surgery section of the American Academy of Ophthalmology's Basic and Clinical Science Course to OpenEvidence and to six general-purpose chatbots. Accuracy was the share of correct answers, compared against results from 1,983 users of the self-assessment program.

Read the study abstract

Purpose To assess the accuracy of OpenEvidence relative to six leading large language models (LLMs) in answering refractive surgery multiple-choice questions, and to compare their performance with users of the American Academy of Ophthalmology (AAO) Basic and Clinical Science Course (BCSC) Self-Assessment Program. Setting Not applicable. Design Randomized questionnaire-based study using 225 questions from the refractive surgery section of the AAO BCSC Program. Methods OpenEvidence and 6 LLMs (ChatGPT-4o, ChatGPT…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 225 · 1 independent sourcePeer-reviewedTRV-2026-2288· cite & verify
OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an Advantage?
A compend of the diseases of the eye and refraction [electronic resource] : including treatment and surgery by Gould, George M. (George Milbry), 1848-1922 Pyle, Walter L. (Walter Lytle), 1871-1921. Public domain

In brief

OpenEvidence, which is built around curated medical evidence, scored lowest at 76% and did not beat the human user benchmark. This matters for clinicians weighing specialty tools against general chatbots, but the authors caution that the result is specific to this multiple-choice format and does not show broader clinical inferiority.

Main points

  1. The test used 225 refractive surgery questions from the AAO BCSC Program.
  2. Compared tools were OpenEvidence, ChatGPT-4o, ChatGPT-5, Claude, Copilot, Gemini and Grok.
  3. OpenEvidence had the lowest accuracy at 76%.
  4. 1,983 BCSC Self-Assessment Program users served as the human benchmark, which OpenEvidence did not exceed.
  5. The authors say the gap was not fully explained by response-format noncompliance.

The problem

The medical-evidence tool OpenEvidence had the lowest accuracy (76%) on 225 refractive surgery questions, below six general-purpose chatbots and the human benchmark of ophthalmology trainees and users.

The rundown

The study was a randomized questionnaire-based comparison in which every tool answered the same set of questions. Accuracy was defined as the proportion of correct responses, allowing direct comparison between the specialized tool and general models.

The authors concluded that OpenEvidence performed significantly worse than all six general-purpose models. They found the difference was not fully explained by failures to follow the required answer format, so the shortfall appears to reflect more than formatting problems.

The authors explicitly warn against reading the result as evidence of globally inferior medical knowledge or poorer performance in open-ended clinical use.

Sources

  1. Peer-reviewedJournal of Cataract & Refractive Surgery2026-10-26

The debate