aceA Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek
Aim The proliferation of clinical practice guidelines presents significant challenges for ensuring evidence-based care, since evaluating their trustworthiness with the Reporting Items for practice Guidelines in HealThcare (RIGHT) criteria is both time-consuming and resource-intensive. To address this, the Artificial intelligence-empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards (ADVANCED) working group developed ad^Writer, an artificial intelligence system, designed to automate guidelin…
ad^Writer, running on two large language models, agreed well with human RIGHT-based assessments of 60 guidelines and could speed up guideline appraisal as a supportive tool.
Agreement between the AI assessments and human evaluations came with wide limits of agreement, and performance varied across individual RIGHT items and between the two models.
Although agreement with human raters was good, the limits of agreement were wide and accuracy varied across RIGHT items, so the tool is presented as support for manual evaluation rather than a replacement.
Evidence
- Peer-reviewedJournal of Evidence-Based Medicine2026-10-06
How should this claim be treated?
Truvace Impact Record TRV-2026-1800, v1: “A Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek.” Truvace, 2026-10-18. /record/TRV-2026-1800 (accessed at citation time). sha256 83c7de80d95db3f0…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1800 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.