LLM responses to consensus-based bone stress injury questions

Source article: Large language models show distinct multidimensional performance profiles and variable reproducibility in consensus-based bone stress injury

Purpose This study evaluated five publicly available large language model-based systems using a multidimensional, expert-based assessment of consensus-based clinical questions on bone stress injuries. Methods Ten standardised questions derived from a 2025 international Delphi consensus on bone stress injuries were submitted to ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Responses were independently rated by clinical experts using a multidimensional evaluation framework covering a…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1749· cite & verify

Trace impact readingContested

P 69

Both sides are scored from claims and sources, not community votes.

G 72

In brief

Researchers put ten standardised questions drawn from a 2025 international Delphi consensus on bone stress injuries to five public AI systems: ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Clinical experts rated each answer on seven dimensions, including accuracy, reasoning, potential harm and trust.

The findings suggest chatbots could help spread consensus-based orthopaedic knowledge, but the models differed from one another and were not equally consistent when questions were repeated. The authors conclude the tools may support, not replace, consensus documents and clinicians. The abstract is truncated, so detailed scores are not available.

Main points

Ten questions from a 2025 Delphi consensus on bone stress injuries were used as the test set.
Experts scored responses on accuracy, comprehensiveness, understanding, reasoning, clarity, potential harm and trust using 5-point scales.
Significant between-model differences appeared for comprehensiveness, reasoning, clarity and trust.
Overall performance was moderate to high, but reproducibility varied across models.

The gain

Publicly available LLM systems showed moderate-to-high performance on expert-derived bone stress injury questions, suggesting they could help communicate consensus-based clinical knowledge.

The problem

Performance varied across evaluation dimensions and in reproducibility, so these models should not replace primary consensus documents or expert clinical judgement.

The rundown

The five systems tested were ChatGPT-5, Perplexity AI, Claude 4, Google Gemini 2.5 and Microsoft Copilot. Median scores per question and model were compared using Kruskal-Wallis tests with Dunn's post hoc tests and Bonferroni correction, and interrater agreement was measured with weighted Gwet's agreement coefficient.

Reproducibility was tested separately by repeatedly submitting nine questions representing no, partial and full expert consensus to each model. Response consistency was scored from 1 to 3.

The authors frame the models as aids for communicating consensus knowledge, with the clear caveat that they should not substitute for primary consensus documents or expert clinical judgement.

What this doesn’t fix

The abstract's results section is truncated, and the authors state that performance and reproducibility varied, so the models cannot replace primary consensus documents or expert clinical judgement.