Truace
Subscribe

AI chatbots gave carpal tunnel advice that often departed from medical guidelines

Study title: Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome

Researchers posed standardized questions drawn from American Academy of Orthopaedic Surgeons carpal tunnel guidelines to four chatbots: ChatGPT-4, Gemini, Claude-3 and Mistral-7B. Each prompt was repeated three times per model, and two blinded reviewers graded the synthesized answers as concordant, discordant or indeterminate.

Read the study abstract

Background Large language models (LLMs) such as ChatGPT, Gemini, Claude, and Mistral are increasingly used to obtain medical information, but their accuracy in carpal tunnel syndrome (CTS) management is uncertain. This study evaluated whether LLM-generated recommendations align with American Academy of Orthopaedic Surgeons (AAOS) clinical practice guidelines (CPGs). Methods Four LLMs (ChatGPT-4, Gemini, Claude-3, Mistral-7B) were queried with standardized prompts derived from AAOS CTS CPGs. Each prompt was asked…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 252 · 1 independent sourcePeer-reviewedTRV-2026-2306· cite & verify
Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
Carpal Tunnel by Maen K. Househ. CC BY-SA 3.0 · https://creativecommons.org/licenses/by-sa/3.0

In brief

The results matter because people increasingly turn to chatbots for medical information, and uneven guideline alignment could mislead patients or clinicians. The study tested only guideline-based prompts and specific model versions, so it does not show how the tools perform in real clinical conversations or how newer versions would fare.

Main points

  1. Across 252 recommendations, 72.6% were concordant, 11.5% discordant and 15.9% indeterminate.
  2. ChatGPT had the highest concordance at 92.1% and the lowest discordance at 1.6%.
  3. Mistral performed worst, with 52.4% concordance and 17.5% discordance.
  4. Reviewer agreement was excellent, with a Cohen's kappa of 0.91.

The problem

More than one in four chatbot recommendations for carpal tunnel syndrome did not match AAOS guidelines, and weaker models such as Mistral-7B performed far worse than ChatGPT-4.

The rundown

The team used standardized prompts derived from AAOS clinical practice guidelines for carpal tunnel syndrome. Four models were tested: ChatGPT-4, Gemini, Claude-3 and Mistral-7B, each queried three times per prompt.

Two blinded reviewers graded responses, resolving disagreements by consensus. Differences in concordance between models were compared with chi-square testing using Bonferroni correction.

The authors conclude that variable performance underscores the need for clinician oversight and careful validation before clinical use.

Sources

  1. Peer-reviewedHAND2026-10-27

The debate