Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1800
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-18T06:55:04.987830Z
status: published
lens: trace
sector: health
headline: A Comparative Study of ad^Writer and Human Evaluations of Guideline Reporting Quality: Performance of ChatGPT and DeepSeek
dek: Aim The proliferation of clinical practice guidelines presents significant challenges for ensuring evidence-based care, since evaluating their trustworthiness with the Reporting Items for practice Guidelines in HealThcare (RIGHT) criteria is both time-consuming and resource-intensive. To address this, the Artificial intelligence-empowered DeVelopment and AssessmeNt of healthCare guidElines and stanDards (ADVANCED) working group developed ad^Writer, an artificial intelligence system, designed to automate guidelin…
gain_title: ad^Writer, running on two large language models, agreed well with human RIGHT-based assessments of 60 guidelines and could speed up guideline appraisal as a supportive tool.
problem_title: Agreement between the AI assessments and human evaluations came with wide limits of agreement, and performance varied across individual RIGHT items and between the two models.
trace_subject: AI assessment of clinical guideline reporting quality using the RIGHT criteria
gain_reading: ad^Writer, running on two large language models, agreed well with human RIGHT-based assessments of 60 guidelines and could speed up guideline appraisal as a supportive tool.
gain_evidence: Both models showed good agreement with human | enhances the efficiency of the guideline appraisal process, saving time and human resources
problem_reading: Agreement between the AI assessments and human evaluations came with wide limits of agreement, and performance varied across individual RIGHT items and between the two models.
problem_evidence: but had wide limits of agreement | however, the results varied
quick_read: Researchers tested ad^Writer, an AI system built to automate checking of how well clinical practice guidelines follow the RIGHT reporting criteria. Sixty guidelines were scored by the tool with two large language models, three runs each, and the results were compared against human evaluations from published systematic reviews.

Appraising guidelines by hand is slow and resource-intensive, so a reliable assistant could help stakeholders judge trustworthiness more efficiently. The wide limits of agreement and item-level variation show the tool is better suited to support manual review than to replace it, and the results come from a single set of 60 guidelines.
limitation: Although agreement with human raters was good, the limits of agreement were wide and accuracy varied across RIGHT items, so the tool is presented as support for manual evaluation rather than a replacement.
tag: Dual reading
key_points: Sixty guidelines from four published systematic reviews were included, with the authors of the original reviews contacted directly. | Each of the two language models performed three independent assessments, combined by majority rule. | DeepSeek had better agreement with human ratings: mean difference -1.40, with 95% limits of agreement from -11.22 to 8.42. | Both models reached at least 75% accuracy on 23 items (65.71%). | ChatGPT was more consistent, with an ICC of 0.93 versus 0.88 for DeepSeek.
rundown: The study was prompted by the growing number of clinical practice guidelines and the time and resources needed to judge them with the RIGHT checklist. The ADVANCED working group developed ad^Writer to automate that assessment.

Methodologically, the AI output was compared with human results from systematic reviews as the reference standard. Agreement, consistency, and stability were examined with Bland-Altman plots, accuracy measures, and intraclass correlation coefficients, focusing on individual RIGHT items.

The two models showed different strengths. DeepSeek had better agreement and accuracy against human ratings, while ChatGPT produced more consistent repeated assessments. The authors conclude the system shows potential for rapid, accurate, and reproducible assessment as a supportive tool.
sources:
- peer_reviewed | Journal of Evidence-Based Medicine | https://doi.org/10.1111/jebm.70193 | 2026-10-06
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

83c7de80d95db3f0c8832ec352a236b07b5761670db4049759f119d5ff878e49

previous

0000000000000000000000000000000000000000000000000000000000000000