Truace
Subscribe

OsteoCHAT: real-world patient evaluation and benchmarking of a guideline-grounded osteoporosis chatbot

Patients with osteoporosis have information needs that routine care inconsistently addresses. In this multicenter real-world evaluation, the LLM-based, guideline-grounded chatbot OsteoCHAT showed high participant acceptability and achieved expert-rated answer quality comparable to general-purpose LLMs. Despite challenges in complex treatment and safety-related topics, guideline-grounded chatbots may represent useful educational adjuncts. Purpose Patients with osteoporosis have information needs that routine care…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1350· cite & verify
OsteoCHAT: real-world patient evaluation and benchmarking of a guideline-grounded osteoporosis chatbot
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

In brief

Researchers built OsteoCHAT, a retrieval-augmented chatbot grounded in the current German osteoporosis guideline, and tested it with patients at seven centers in Germany. Participants asked questions, rated individual answers, and completed a questionnaire on usability, trust and usefulness. The tool was also benchmarked against Gemini, ChatGPT and Meta AI by five clinical experts.

The findings suggest guideline-grounded chatbots could serve as educational adjuncts where routine care leaves information gaps. However, trust was lower than other measures and every system showed weaker safety communication, so such tools are not a substitute for clinical advice. The study reports acceptability and expert-rated answer quality, not clinical outcomes.

Main points

  1. The evaluation recorded 1,075 question-answer interactions across seven German centers.
  2. Of 389 individually rated responses, 97.4% received positive feedback.
  3. Among 217 complete questionnaires, 84.3% found answers easily understandable and 77.2% perceived time savings.
  4. Trust was comparatively lower, at 64.7% agreement.
  5. Median expert scores were 14/15 for Gemini, ChatGPT and OsteoCHAT and 13/15 for Meta AI.

The problem

Expert reviewers found OsteoCHAT and the other LLMs weak on pharmacological treatment indications and on communicating adverse and rare safety-critical events.

The rundown

OsteoCHAT uses retrieval-augmented generation anchored to the current German osteoporosis guideline. Participants interacted with it, gave real-time feedback on responses, and completed a standardized questionnaire that also asked about preference over conventional internet search.

For benchmarking, ten frequently asked questions derived from the BfO patient guideline served as the gold standard. Five clinical experts independently rated blinded responses from four systems across five domains, each scored 0-3.

Usability results were favorable: 79.8% found OsteoCHAT easy to use and 78.7% considered it a useful addition to existing educational materials. Experts nonetheless flagged weaknesses in pharmacological treatment indication and in communicating adverse and rare safety-critical events across all examined systems.

Sources

  1. Peer-reviewedOsteoporosis International2026-10-03

The debate