Truace
Subscribe
52/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Study finds ChatGPT-5.1 badly missed 3 of 7 orthopaedic meta-analysis results

Study title: Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics

Evidence type
Benchmark or lab evaluation
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2760cite or check it

Study title: Large language models are comparable with commonly used statistical software: A validation of GPT 5.1 for frequentist meta‐analysis in orthopaedics

A methodological comparison study tested whether ChatGPT-5.1 could reproduce meta-analysis calculations as well as standard statistical software. Across seven outcomes from two published orthopaedic meta-analyses, the chatbot got the direction of every effect right, but three outcomes (43%) deviated majorly from reference results, one (14%) moderately and three (43%) only slightly.

Read the study abstract

Abstract Purpose The purpose of this study was to evaluate whether Chat Generative Pre‐trained Transformer (ChatGPT; Version 5.1) can reproduce frequentist meta‐analytic calculations with an accuracy comparable to established statistical software in orthopaedic research. Methods In this methodological comparison study, data from two previously published orthopaedic meta‐analyses with identical statistical architectures as reference standards were used. Between‐study variance ( τ 2 ) was estimated using the Sidik…

The problem

In a comparison using data from two published orthopaedic meta-analyses, ChatGPT-5.1 deviated majorly from R-package reference results in three of seven outcomes, mostly where studies varied widely.

In brief

The errors clustered where the underlying studies disagreed most. Agreement was best in low-heterogeneity settings, while large deviations appeared in outcomes with pronounced between-study heterogeneity, particularly under random-effects models, and the model tended to underestimate between-study variability.

The authors conclude that large language models may support exploratory analyses but cannot fully replace dedicated statistical software for meta-analyses in orthopaedic research, a point that matters for researchers tempted to hand pooled calculations to a chatbot.

Main points

  1. The study used data extraction tables from two previously published orthopaedic meta-analyses as reference standards.
  2. Reference results came from the meta and metafor packages in R, using the Sidik–Jonkman method for between-study variance and the Hartung–Knapp adjustment for random-effects models.
  3. ChatGPT-5.1 reproduced the direction of effects in all seven evaluated outcomes.
  4. Deviations were minor in three outcomes (43%), moderate in one (14%) and major in three (43%).
  5. Substantial errors occurred where between-study heterogeneity was pronounced, especially under random-effects models.

The full story

Researchers gave ChatGPT-5.1 the original data extraction tables from two published orthopaedic meta-analyses and told it to repeat the same analyses. It had to produce pooled mean differences, confidence intervals and heterogeneity statistics (I², τ², p values), which were then checked against verified results from the meta and metafor packages in R.

The random-effects models relied on the Sidik–Jonkman estimator for between-study variance and the Hartung–Knapp adjustment for uncertainty, while common-effect models assume τ² = 0. The chatbot's weakest results came in random-effects scenarios with highly varied studies, where it tended to underestimate between-study variability.

The paper is rated Level III evidence and describes the chatbot as showing emerging capability in low-heterogeneity settings, but not as reliable enough to work as a standalone tool.

Sources

1 on file
  1. Knee Surgery, Sports Traumatology, Arthroscopydoi:10.1002/ksa.70379
    Peer-reviewed2026-03-11

Talk it over

Four questions to start onWho carries this cost, and what would have to change to prevent it?

Start with a question

Study finds ChatGPT-5.1 badly missed 3 of 7 orthopaedic meta-analysis resultshttps://truvace.com/item/large-language-models-are-comparable-with-commonly-used-statistical-software-a-v

  1. Who carries this cost, and what would have to change to prevent it?

    The problem In a comparison using data from two published orthopaedic meta-analyses, ChatGPT-5.1 deviated majorly from R-package reference results in three of seven outcomes, mostly where studies varied widely.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a benchmark or lab test, from Knee Surgery, Sports Traumatology, Arthroscopy.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The comparison used data from only two previously published orthopaedic meta-analyses with identical statistical architectures, covering seven outcomes, so the findings may not extend to other analysis types.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Science