Truace
Subscribe

AI assessment of clinical guideline reporting quality using the RIGHT criteria

Source article: A Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek

Aim The proliferation of clinical practice guidelines presents significant challenges for ensuring evidence-based care, since evaluating their trustworthiness with the Reporting Items for practice Guidelines in HealThcare (RIGHT) criteria is both time-consuming and resource-intensive. To address this, the Artificial intelligence-empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards (ADVANCED) working group developed ad^Writer, an artificial intelligence system, designed to automate guidelin…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1800· cite & verify
A Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek
Using clinical practice guidelines to evaluate quality of care by Schoenbaum, Stephen C Sundwall, David N United States. Agency for Health Care Policy and Research. Office of the Forum for Quality and Effectiveness in Health Care. Public domain
Trace impact readingContested
P 67The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 65The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers tested ad^Writer, an AI system built to automate checking of how well clinical practice guidelines follow the RIGHT reporting criteria. Sixty guidelines were scored by the tool with two large language models, three runs each, and the results were compared against human evaluations from published systematic reviews.

Appraising guidelines by hand is slow and resource-intensive, so a reliable assistant could help stakeholders judge trustworthiness more efficiently. The wide limits of agreement and item-level variation show the tool is better suited to support manual review than to replace it, and the results come from a single set of 60 guidelines.

Main points

  1. Sixty guidelines from four published systematic reviews were included, with the authors of the original reviews contacted directly.
  2. Each of the two language models performed three independent assessments, combined by majority rule.
  3. DeepSeek had better agreement with human ratings: mean difference -1.40, with 95% limits of agreement from -11.22 to 8.42.
  4. Both models reached at least 75% accuracy on 23 items (65.71%).
  5. ChatGPT was more consistent, with an ICC of 0.93 versus 0.88 for DeepSeek.

The gain

ad^Writer, running on two large language models, agreed well with human RIGHT-based assessments of 60 guidelines and could speed up guideline appraisal as a supportive tool.

The problem

Agreement between the AI assessments and human evaluations came with wide limits of agreement, and performance varied across individual RIGHT items and between the two models.

The rundown

The study was prompted by the growing number of clinical practice guidelines and the time and resources needed to judge them with the RIGHT checklist. The ADVANCED working group developed ad^Writer to automate that assessment.

Methodologically, the AI output was compared with human results from systematic reviews as the reference standard. Agreement, consistency, and stability were examined with Bland-Altman plots, accuracy measures, and intraclass correlation coefficients, focusing on individual RIGHT items.

The two models showed different strengths. DeepSeek had better agreement and accuracy against human ratings, while ChatGPT produced more consistent repeated assessments. The authors conclude the system shows potential for rapid, accurate, and reproducible assessment as a supportive tool.

What this doesn’t fix

Although agreement with human raters was good, the limits of agreement were wide and accuracy varied across RIGHT items, so the tool is presented as support for manual evaluation rather than a replacement.

Sources

  1. Peer-reviewedJournal of Evidence-Based Medicine2026-10-06

The debate