Truace
Subscribe

OpenEvidence-generated eConsults for MASLD

Source article: Large Language Model-Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations

Objective: To evaluate whether a context-specific large language model, OpenEvidence, can generate gastroenterology/hepatology eConsult responses for metabolic dysfunction-associated steatotic liver disease (MASLD), while characterizing the frequency and potential severity of errors. Patients and methods: In this retrospective cohort study at a tertiary academic medical center, 125 specialist electronic consultations (eConsults) for MASLD (from January 2020 to December 2024) were paired with OpenEvidence-generat…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-2035· cite & verify
Large Language Model-Generated Electronic Consultations: A Blinded Comparative Study of OpenEvidence and Gastroenterologist Electronic Consultations
Navy Medical Newsletter Vol. 55 No. 4 April 1970 by U.S. Navy. Bureau of Medicine and Surgery. Public domain
Trace impact readingNegative state
P 69The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 64The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers at a tertiary academic medical center paired 125 past specialist eConsults on metabolic dysfunction-associated steatotic liver disease with responses produced by OpenEvidence from standardized prompts. Five physician reviewers, blinded to the source, rated all responses with a previously validated instrument.

The AI responses were rated higher in perceived quality and informational adequacy, which suggests a possible role in assisting protocol-driven specialist consultations. However, the ratings measure perceived quality rather than patient outcomes, and the presence of one potentially major error shows that oversight remains a open question.

Main points

  1. The mean difference in eConsult Specialist Quality of Response scores favored OpenEvidence by 6.4 points (95% CI, 6.0-6.9).
  2. Global quality ratings favored OpenEvidence with an odds ratio of 16.1.
  3. OpenEvidence responses were more often judged to give an appropriate amount of information, 70.9% versus 33.9%.
  4. Of 125 OpenEvidence eConsults, 12 had minor errors and 1 had a potentially major error.

The gain

In blinded ratings of 125 paired MASLD eConsults, responses generated by the OpenEvidence language model scored higher on quality and on appropriate information amount than specialist-written responses.

The problem

Among the 125 OpenEvidence-generated MASLD eConsults, 12 contained minor errors and 1 contained a potentially major error, so clinically relevant mistakes still occurred.

The rundown

The study was a retrospective cohort at a single tertiary academic medical center, using specialist eConsults for MASLD written between January 2020 and December 2024. Each was matched with an OpenEvidence-generated version.

Five blinded physician reviewers scored every eConsult. A separate board-certified gastroenterologist reviewed the OpenEvidence outputs to identify errors and assess their potential for harm.

The authors conclude that the findings support further evaluation of OpenEvidence-assisted eConsult workflows for protocol-driven conditions, rather than replacing specialists outright.

What this doesn’t fix

The study was a retrospective, single-center comparison of perceived quality for one condition, and the AI outputs still included errors, one potentially major.

Sources

  1. Peer-reviewedMayo Clinic Proceedings: Digital Health2026-09-18

The debate