Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1749
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-17T22:02:32.322055Z
status: published
lens: trace
sector: health
headline: Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury
dek: Purpose This study evaluated five publicly available large language model-based systems using a multidimensional, expert-based assessment of consensus-based clinical questions on bone stress injuries. Methods Ten standardised questions derived from a 2025 international Delphi consensus on bone stress injuries were submitted to ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Responses were independently rated by clinical experts using a multidimensional evaluation framework covering a…
gain_title: Publicly available LLM systems showed moderate-to-high performance on expert-derived bone stress injury questions, suggesting they could help communicate consensus-based clinical knowledge.
problem_title: Performance varied across evaluation dimensions and in reproducibility, so these models should not replace primary consensus documents or expert clinical judgement.
trace_subject: LLM responses to consensus-based bone stress injury questions
gain_reading: Publicly available LLM systems showed moderate-to-high performance on expert-derived bone stress injury questions, suggesting they could help communicate consensus-based clinical knowledge.
gain_evidence: demonstrated moderate-to-high performance in addressing expert-derived, consensus-based questions on bone stress injuries
problem_reading: Performance varied across evaluation dimensions and in reproducibility, so these models should not replace primary consensus documents or expert clinical judgement.
problem_evidence: performance varied across evaluation dimensions and reproducibility
quick_read: Researchers put ten standardised questions drawn from a 2025 international Delphi consensus on bone stress injuries to five public AI systems: ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Clinical experts rated each answer on seven dimensions, including accuracy, reasoning, potential harm and trust.

The findings suggest chatbots could help spread consensus-based orthopaedic knowledge, but the models differed from one another and were not equally consistent when questions were repeated. The authors conclude the tools may support, not replace, consensus documents and clinicians. The abstract is truncated, so detailed scores are not available.
limitation: The abstract's results section is truncated, and the authors state that performance and reproducibility varied, so the models cannot replace primary consensus documents or expert clinical judgement.
tag: Dual reading
key_points: Ten questions from a 2025 Delphi consensus on bone stress injuries were used as the test set. | Experts scored responses on accuracy, comprehensiveness, understanding, reasoning, clarity, potential harm and trust using 5-point scales. | Significant between-model differences appeared for comprehensiveness, reasoning, clarity and trust. | Overall performance was moderate to high, but reproducibility varied across models.
rundown: The five systems tested were ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Median scores per question and model were compared using Kruskal-Wallis tests with Dunn's post hoc tests and Bonferroni correction, and interrater agreement was measured with weighted Gwet's agreement coefficient.

Reproducibility was tested separately by repeatedly submitting nine questions representing no, partial and full expert consensus to each model. Response consistency was scored from 1 to 3.

The authors frame the models as aids for communicating consensus knowledge, with the clear caveat that they should not substitute for primary consensus documents or expert clinical judgement.
sources:
- peer_reviewed | Journal of Experimental Orthopaedics | https://doi.org/10.1002/jeo2.70939 | 2026-10-07
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

68413e2795ebeef80181962d46d22cccf57c54d435c80be839d8941cb2bb01c2

previous

0000000000000000000000000000000000000000000000000000000000000000