aceDevelopment and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology
Patients with rheumatic diseases have persistent information needs that are not fully addressed in routine care. We developed and evaluated guideline-grounded, large language model (LLM) chatbots to support patient self-management and education in rheumatology.Ten disease-specific chatbots based on German guidelines were co-developed and deployed through 13 rheumatology centres and six patient organisations. Chatbot users rated responses and completed a questionnaire. User questions, feedback, and response chara…
Patients using guideline-grounded rheumatology chatbots reported high usability and satisfaction, with most answers rated safe and correct in real-world deployment.
Chatbots failed to answer 4.2% of interactions, received negative ratings for insufficient detail, and showed only 45% full guideline adherence with weak agreement between LLM and physician assessments.
Findings are limited by reliance on LLM-based quality ratings with weak agreement with physician assessment and lack of measured educational effectiveness or independent clinical safety validation.
Evidence
- Peer-reviewedJournal of Medical Systems2026-10-05
How should this claim be treated?
Truvace Impact Record TRV-2026-1292, v1: “Development and Nationwide Multicentre Evaluation of Guideline-Grounded Large Language Model Chatbots to Support Patient Self-Management and Education in Rheumatology.” Truvace, 2026-10-06. /record/TRV-2026-1292 (accessed at citation time). sha256 ff14a54f27f6b02d…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1292 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.