Medical literature-specific AI tools answering lacrimal drainage disorder questions

Source article: Medical Literature-Specific Artificial Intelligence: Analyzing the performance of OpenEvidence, Heidi Evidence, Consensus, and PubMed.ai in the context of Lacrimal Drainage Disorders

Purpose This study aimed to report the performance of medical literature-specific artificial intelligence (MLS-AI) deep learning tools in the context of lacrimal drainage disorders. Methods Prompt engineering was performed using 22 previously validated questions and statements to include common and rare aspects of lacrimal drainage disorders. Prompts were presented at least twice to the latest versions of OpenEvidence, Heidi Evidence, Consensus, and PubMed.ai [Accessed May 14-26, 2026]. The prompts included gene…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1740· cite & verify

Trace impact readingNegative state

P 71

Both sides are scored from claims and sources, not community votes.

G 64

In brief

Researchers tested four medical literature-specific AI tools, OpenEvidence, Heidi Evidence, Consensus and PubMed.ai, using 22 validated prompts on lacrimal drainage disorders, including general, rare and controversial topics. The authors graded each response as correct, partially correct or factually incorrect.

The results show that tools marketed for the same purpose can differ greatly in dependability, which matters for clinicians who may consult them for specialty questions. The study covers one narrow ophthalmic topic, and the authors conclude that all responses currently need human oversight and verification.

Main points

Twenty-two previously validated questions and statements covered common and rare aspects of lacrimal drainage disorders.
OpenEvidence was correct on 90.9% of prompts, versus 63.6% for Heidi and 54.5% for Consensus.
PubMed.ai performed worst, with 22.7% factually incorrect answers and two prompts unanswered.
The authors say the tools could complement one another depending on context and prompt type.

The gain

OpenEvidence answered 20 of 22 lacrimal drainage disorder prompts correctly and none incorrectly, suggesting some literature-specific AI tools can give largely reliable specialty answers.

The problem

PubMed.ai gave factually incorrect answers to 5 of 22 prompts and no response to 2, and Heidi hallucinated twice, showing some tools produce unreliable clinical information.

The rundown

Prompts were presented at least twice to the latest versions of each tool, accessed May 14-26, 2026. They ranged from managing congenital nasolacrimal duct obstructions to terminology such as idiopathic canalicular inflammatory disease, and included controversial topics such as silicone intubation and mitomycin-C in dacryocystorhinostomy.

Responses were assessed for quality, organization, clarity, evidence-based content, updated knowledge, specificity, speed and factual inaccuracies. Heidi was graded 63.6% correct, 31.8% partially correct and 4.6% incorrect, while Consensus was 54.5% correct, 40.9% partially correct and 4.6% incorrect.

The authors rank OpenEvidence slightly ahead of Heidi, with Consensus a close third and PubMed.ai far behind. They say the tools need further training to capture more indexed mainstream journal publications.

What this doesn’t fix

The authors graded the responses themselves on only 22 prompts in one ophthalmic subtopic, and they state that all responses currently need human oversight and verification before being accepted as true.