Truace
Subscribe

Chatbot answers to patient questions about osteoporotic vertebral compression fractures

Source article: Performance of four large language model chatbots in responding to patient-facing questions about osteoporotic vertebral compression fractures: a cross-sectional paired evaluation

Background Large language model (LLM) chatbots are increasingly used by the public to obtain health information, but the safety and quality of their responses to questions about osteoporotic vertebral compression fractures (OVCFs) remain unclear. Objective To compare the safety and patient-facing response quality of four publicly accessible LLM chatbots when responding to standardized questions about osteoporotic vertebral compression fractures. Methods In this cross-sectional paired evaluation, 36 standardized…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1715· cite & verify
Performance of four large language model chatbots in responding to patient-facing questions about osteoporotic vertebral compression fractures: a cross-sectional paired evaluation
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.
Trace impact readingContested
P 70The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 69The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers put 36 standardized patient-style questions on osteoporotic vertebral compression fractures to ChatGPT, Gemini, DeepSeek and Doubao on June 20, 2026, then repeated the exercise a week later. Five clinicians blinded to chatbot identity rated the 288 responses for safety, accuracy, empathy, information quality, transparency and readability.

The results matter because the public increasingly turns to chatbots for health information, and accurate-looking answers can still contain unsafe content. The study cannot show that the safety differences between chatbots are real, and its findings apply to one location, one language, single-turn queries and two dates, so they may not hold after model updates.

Main points

  1. Potentially unsafe responses appeared for all four chatbots: 3 of 36 for ChatGPT, 5 of 36 for DeepSeek, 6 of 36 for Gemini and 7 of 36 for Doubao.
  2. The overall difference in safety between chatbots was not statistically significant (Cochran's Q = 3.621, P = 0.305).
  3. Factual accuracy differed only modestly among chatbots, while empathy varied far more, with ChatGPT and Gemini scoring higher than DeepSeek and Doubao.
  4. Doubao's responses had the greatest estimated textual complexity among the four chatbots.
  5. The authors concluded that chatbot-generated information on these fractures needs clinician oversight and further validation.

The gain

Clinician raters found the chatbots' answers to patient questions about osteoporotic vertebral compression fractures generally factually accurate, and ChatGPT and Gemini scored higher than DeepSeek and Doubao on empathy.

The problem

Every one of the four chatbots produced some responses that raters classified as potentially unsafe for patients asking about osteoporotic vertebral compression fractures, so the tools are not safe as standalone patient education.

The rundown

The 36 questions covered symptoms and warning signs, diagnosis and imaging, conservative management, medication safety, rehabilitation, vertebral augmentation, fracture prevention and special clinical scenarios. They were submitted in English in independent single-turn sessions from Jiangsu, China, which produced 144 responses for the main comparison.

Rater agreement was high, with a Fleiss' kappa of 0.818 for safety and intraclass correlations of about 0.82 for the other measures. Differences among chatbots on DISCERN, EQIP, the JAMA benchmark and the Global Quality Score were all statistically significant (P < 0.001).

A second generation round 7 days later gave 144 more responses to test stability. Safety agreement between rounds ranged from 88.9% to 97.2%, and median ICC values across 12 non-safety outcomes ranged from 0.781 to 0.904, so output was fairly consistent but still varied between generations.

What this doesn’t fix

The findings are specific to the access location, language, single-turn conditions and two generation dates, and should not be assumed to hold across settings or after model updates; stability across rounds was good but not complete.

Sources

  1. Peer-reviewedFrontiers in Musculoskeletal Disorders2026-10-16

The debate