Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1715
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-17T18:12:54.483788Z
status: published
lens: trace
sector: health
headline: Performance of four large language model chatbots in responding to patient-facing questions about osteoporotic vertebral compression fractures: a cross-sectional paired evaluation
dek: Background Large language model (LLM) chatbots are increasingly used by the public to obtain health information, but the safety and quality of their responses to questions about osteoporotic vertebral compression fractures (OVCFs) remain unclear. Objective To compare the safety and patient-facing response quality of four publicly accessible LLM chatbots when responding to standardized questions about osteoporotic vertebral compression fractures. Methods In this cross-sectional paired evaluation, 36 standardized…
gain_title: Clinician raters found the chatbots' answers to patient questions about osteoporotic vertebral compression fractures generally factually accurate, and ChatGPT and Gemini scored higher than DeepSeek and Doubao on empathy.
problem_title: Every one of the four chatbots produced some responses that raters classified as potentially unsafe for patients asking about osteoporotic vertebral compression fractures, so the tools are not safe as standalone patient education.
trace_subject: Chatbot answers to patient questions about osteoporotic vertebral compression fractures
gain_reading: Clinician raters found the chatbots' answers to patient questions about osteoporotic vertebral compression fractures generally factually accurate, and ChatGPT and Gemini scored higher than DeepSeek and Doubao on empathy.
gain_evidence: generally high clinician-rated factual accuracy | higher scores for ChatGPT and Gemini than for DeepSeek and Doubao
problem_reading: Every one of the four chatbots produced some responses that raters classified as potentially unsafe for patients asking about osteoporotic vertebral compression fractures, so the tools are not safe as standalone patient education.
problem_evidence: all four chatbot interfaces produced some responses classified as potentially unsafe | should not be used as a standalone patient education resource
quick_read: Researchers put 36 standardized patient-style questions on osteoporotic vertebral compression fractures to ChatGPT, Gemini, DeepSeek and Doubao on June 20, 2026, then repeated the exercise a week later. Five clinicians blinded to chatbot identity rated the 288 responses for safety, accuracy, empathy, information quality, transparency and readability.

The results matter because the public increasingly turns to chatbots for health information, and accurate-looking answers can still contain unsafe content. The study cannot show that the safety differences between chatbots are real, and its findings apply to one location, one language, single-turn queries and two dates, so they may not hold after model updates.
limitation: The findings are specific to the access location, language, single-turn conditions and two generation dates, and should not be assumed to hold across settings or after model updates; stability across rounds was good but not complete.
tag: Dual reading
key_points: Potentially unsafe responses appeared for all four chatbots: 3 of 36 for ChatGPT, 5 of 36 for DeepSeek, 6 of 36 for Gemini and 7 of 36 for Doubao. | The overall difference in safety between chatbots was not statistically significant (Cochran's Q = 3.621, P = 0.305). | Factual accuracy differed only modestly among chatbots, while empathy varied far more, with ChatGPT and Gemini scoring higher than DeepSeek and Doubao. | Doubao's responses had the greatest estimated textual complexity among the four chatbots. | The authors concluded that chatbot-generated information on these fractures needs clinician oversight and further validation.
rundown: The 36 questions covered symptoms and warning signs, diagnosis and imaging, conservative management, medication safety, rehabilitation, vertebral augmentation, fracture prevention and special clinical scenarios. They were submitted in English in independent single-turn sessions from Jiangsu, China, which produced 144 responses for the main comparison.

Rater agreement was high, with a Fleiss' kappa of 0.818 for safety and intraclass correlations of about 0.82 for the other measures. Differences among chatbots on DISCERN, EQIP, the JAMA benchmark and the Global Quality Score were all statistically significant (P < 0.001).

A second generation round 7 days later gave 144 more responses to test stability. Safety agreement between rounds ranged from 88.9% to 97.2%, and median ICC values across 12 non-safety outcomes ranged from 0.781 to 0.904, so output was fairly consistent but still varied between generations.
sources:
- peer_reviewed | Frontiers in Musculoskeletal Disorders | https://doi.org/10.3389/fmscd.2026.1935149 | 2026-10-16
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

673f361f0ee1e28fd49294789246bc69f63c532c731e59f21c960e996b16d62a

previous

0000000000000000000000000000000000000000000000000000000000000000