Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2288
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-27T18:34:46.634736Z
status: published
lens: p_space
sector: health
headline: OpenEvidence versus Six Leading Large Language Models in Refractive Surgery: Does Curated Medical Evidence Confer an Advantage?
dek: Purpose To assess the accuracy of OpenEvidence relative to six leading large language models (LLMs) in answering refractive surgery multiple-choice questions, and to compare their performance with users of the American Academy of Ophthalmology (AAO) Basic and Clinical Science Course (BCSC) Self-Assessment Program. Setting Not applicable. Design Randomized questionnaire-based study using 225 questions from the refractive surgery section of the AAO BCSC Program. Methods OpenEvidence and 6 LLMs (ChatGPT-4o, ChatGPT…
gain_title: (none)
problem_title: The medical-evidence tool OpenEvidence had the lowest accuracy (76%) on 225 refractive surgery questions, below six general-purpose chatbots and the human benchmark of ophthalmology trainees and users.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: The medical-evidence tool OpenEvidence had the lowest accuracy (76%) on 225 refractive surgery questions, below six general-purpose chatbots and the human benchmark of ophthalmology trainees and users.
problem_evidence: OpenEvidence achieved the lowest accuracy (76%)
quick_read: Researchers gave 225 multiple-choice questions from the refractive surgery section of the American Academy of Ophthalmology's Basic and Clinical Science Course to OpenEvidence and to six general-purpose chatbots. Accuracy was the share of correct answers, compared against results from 1,983 users of the self-assessment program.

OpenEvidence, which is built around curated medical evidence, scored lowest at 76% and did not beat the human user benchmark. This matters for clinicians weighing specialty tools against general chatbots, but the authors caution that the result is specific to this multiple-choice format and does not show broader clinical inferiority.
limitation: The authors state the task-specific result should not be interpreted as direct evidence of globally inferior medical knowledge or of performance in open-ended clinical applications.
tag: Evidence-backed problem
key_points: The test used 225 refractive surgery questions from the AAO BCSC Program. | Compared tools were OpenEvidence, ChatGPT-4o, ChatGPT-5, Claude, Copilot, Gemini and Grok. | OpenEvidence had the lowest accuracy at 76%. | 1,983 BCSC Self-Assessment Program users served as the human benchmark, which OpenEvidence did not exceed. | The authors say the gap was not fully explained by response-format noncompliance.
rundown: The study was a randomized questionnaire-based comparison in which every tool answered the same set of questions. Accuracy was defined as the proportion of correct responses, allowing direct comparison between the specialized tool and general models.

The authors concluded that OpenEvidence performed significantly worse than all six general-purpose models. They found the difference was not fully explained by failures to follow the required answer format, so the shortfall appears to reflect more than formatting problems.

The authors explicitly warn against reading the result as evidence of globally inferior medical knowledge or poorer performance in open-ended clinical use.
sources:
- peer_reviewed | Journal of Cataract & Refractive Surgery | https://doi.org/10.1097/j.jcrs.0000000000002088 | 2026-10-26
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

08f70fce384cda28f18f40a681111e6bffbe6b063372dbe9ff8d288c1fb580ce

previous

0000000000000000000000000000000000000000000000000000000000000000