Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2306
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-29T06:46:05.202025Z
status: published
lens: p_space
sector: health
headline: Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
dek: Background Large language models (LLMs) such as ChatGPT, Gemini, Claude, and Mistral are increasingly used to obtain medical information, but their accuracy in carpal tunnel syndrome (CTS) management is uncertain. This study evaluated whether LLM-generated recommendations align with American Academy of Orthopaedic Surgeons (AAOS) clinical practice guidelines (CPGs). Methods Four LLMs (ChatGPT-4, Gemini, Claude-3, Mistral-7B) were queried with standardized prompts derived from AAOS CTS CPGs. Each prompt was asked…
gain_title: (none)
problem_title: More than one in four chatbot recommendations for carpal tunnel syndrome did not match AAOS guidelines, and weaker models such as Mistral-7B performed far worse than ChatGPT-4.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: More than one in four chatbot recommendations for carpal tunnel syndrome did not match AAOS guidelines, and weaker models such as Mistral-7B performed far worse than ChatGPT-4.
problem_evidence: Over 1 in 4 LLM-generated CTS recommendations were not concordant with AAOS CPGs. | Mistral performed poorest (52.4% concordance, 17.5% discordance).
quick_read: Researchers posed standardized questions drawn from American Academy of Orthopaedic Surgeons carpal tunnel guidelines to four chatbots: ChatGPT-4, Gemini, Claude-3 and Mistral-7B. Each prompt was repeated three times per model, and two blinded reviewers graded the synthesized answers as concordant, discordant or indeterminate.

The results matter because people increasingly turn to chatbots for medical information, and uneven guideline alignment could mislead patients or clinicians. The study tested only guideline-based prompts and specific model versions, so it does not show how the tools perform in real clinical conversations or how newer versions would fare.
limitation: The authors say performance varied across models and that careful validation and clinician oversight are needed before clinical use, so the results do not establish safety for real-world care.
tag: Evidence-backed problem
key_points: Across 252 recommendations, 72.6% were concordant, 11.5% discordant and 15.9% indeterminate. | ChatGPT had the highest concordance at 92.1% and the lowest discordance at 1.6%. | Mistral performed worst, with 52.4% concordance and 17.5% discordance. | Reviewer agreement was excellent, with a Cohen's kappa of 0.91.
rundown: The team used standardized prompts derived from AAOS clinical practice guidelines for carpal tunnel syndrome. Four models were tested: ChatGPT-4, Gemini, Claude-3 and Mistral-7B, each queried three times per prompt.

Two blinded reviewers graded responses, resolving disagreements by consensus. Differences in concordance between models were compared with chi-square testing using Bonferroni correction.

The authors conclude that variable performance underscores the need for clinician oversight and careful validation before clinical use.
sources:
- peer_reviewed | HAND | https://doi.org/10.1177/15589447261495072 | 2026-10-27
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

c3647bc03bf7db7e1d595b78e81dcdd32754d93281d517d077ed29ba191fa8af

previous

0000000000000000000000000000000000000000000000000000000000000000