Truace
Subscribe

Generative AI chatbot responses to osteoporosis health counseling questions

Source article: Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross-Sectional Comparative Study

Background and aims: Generative artificial intelligence chatbots are increasingly used for health information, but their reliability, readability, and safety in osteoporosis counseling remain uncertain. This study compared four widely available chatbots under standardized conditions. Methods: Twenty-five standardized osteoporosis questions were submitted once to ChatGPT-5, Gemini 2.5 Pro, Copilot, and Perplexity Pro, generating 100 responses. Two model-blinded specialists evaluated the final responses using DISC…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1440· cite & verify
Reliability, Readability, and Safety of Generative AI Chatbots for Osteoporosis Health Counseling: A Multidimensional Cross-Sectional Comparative Study
Avaamo Chatbot by Avaamo. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0
Trace impact readingNegative state
P 73The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 65The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers put the same 25 standardized osteoporosis questions to ChatGPT-5, Gemini 2.5 Pro, Copilot and Perplexity Pro, producing 100 responses. Two specialists who did not know which model wrote each answer scored them with several quality and safety instruments, and automated indices measured readability.

The results suggest that the chatbots people use for health information differ in answer quality, and that the better-scoring answers were also harder for ordinary readers to understand. Safety reviewers still found omissions and an unsupported claim, so the findings do not show that any of these tools is safe to rely on for patient counseling.

Main points

  1. DISCERN scores differed significantly across the four models (p < 0.001; Kendall's W = 0.82).
  2. JAMA benchmark scores did not differ significantly between models (p = 0.63).
  3. Perplexity Pro had the highest median Flesch-Kincaid Grade Level (16.18); Copilot had the lowest (9.49).
  4. All six readability indices differed across models, and every model exceeded the sixth-grade benchmark.

The gain

Perplexity Pro and Copilot earned higher DISCERN information-quality medians (55.0 and 54.0) than Gemini 2.5 Pro (44.0) and ChatGPT-5 (42.0) on osteoporosis questions.

The problem

All four chatbots wrote above the sixth-grade readability benchmark, and the safety review found clinically relevant omissions and an unsupported claim in the answers.

The rundown

Each of the 25 questions was submitted once to each of the four chatbots. Two specialists blinded to the model scored the final responses with DISCERN, EQIP, the Global Quality Scale, JAMA benchmarks and the NOHARM safety framework.

Because every question was answered by all four models, the authors compared outcomes with Friedman tests, followed by paired Wilcoxon signed-rank tests with Holm adjustment. The study was published in 2026 in Health Science Reports.

The authors link the higher information-quality scores to retrieval-enabled systems, which also produced more difficult text. They say this pattern is specific to the tested set and does not establish a general effect of retrieval augmentation.

What this doesn’t fix

The findings apply only to the tested models, prompts and evaluation period, and the authors say they do not establish a universal effect of retrieval augmentation.

Sources

  1. Peer-reviewedHealth Science Reports2026-10-10

The debate