Truace
Subscribe

AI chatbots score moderately on orthopedic exam questions but struggle with hard ones

Study title: Performance of AI Chatbots in Adult Reconstruction Education: A Comparative Evaluation Using AAOS ResStudy Questions

Researchers gave 55 multiple-choice adult reconstruction questions from the AAOS ResStudy bank to seven chatbots, including several ChatGPT versions, Gemini, Claude and Co-Pilot. They recorded accuracy, response time and whether the cited sources existed and were relevant.

Read the study abstract

Background: Artificial intelligence (AI) chatbots are increasingly incorporated into orthopedic education, yet their ability to handle complex, clinically oriented questions remains unclear. This study evaluates the accuracy, response time, and citation reliability of contemporary AI chatbots when answering adult reconstruction questions from the AAOS ResStudy question bank. Methods: Fifty-five multiple-choice adult reconstruction questions were extracted and categorized by anatomic region (hip vs. knee), concep…

By the Truvace desk1 min read1 sourceEvidence: Emerging · 48/100 — Benchmark or lab evaluation · n = 55 · 1 independent sourcePeer-reviewedTRV-2026-2294· cite & verify
Performance of AI Chatbots in Adult Reconstruction Education: A Comparative Evaluation Using AAOS ResStudy Questions
Physical therapy equipment (MAMAS FB45-23-6), National Museum of Health and Medicine (302824654) by National Museum of Health and Medicine. CC BY 2.0 · https://creativecommons.org/licenses/by/2.0
Trace impact readingNegative state
P 56The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 49The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

The results suggest chatbots can help with descriptive orthopedic knowledge but are not reliable enough to serve as a primary study resource. Performance fell sharply as questions got harder, and the study tested only one question bank, so how this carries over to real training or clinical decisions is unclear.

Main points

  1. Overall accuracy across the seven chatbots was 67.5%.
  2. GPT-4o scored 80%, ahead of AAOS members at 75%; Co-Pilot ranked second at 70.9%.
  3. Accuracy was highest on clinical presentation (78.6%) and lowest on prognosis (63.1%).
  4. Average response time was 3.6 seconds, with no significant correlation between speed and accuracy.

The gain

GPT-4o answered 80% of adult reconstruction questions correctly, above the 75% score of AAOS members, and Co-Pilot's citations were all real and relevant.

The problem

Overall chatbot accuracy was only 67.5%, falling to 45.4% on difficult questions and 63.1% on prognosis, so chatbots struggle with applied reasoning.

The rundown

The 55 questions were categorized by anatomic region (hip or knee), concept (clinical presentation, prognosis, management) and AAOS-defined difficulty. Statistical tests included chi-square, t-tests, Pearson correlation, ANOVA and Kruskal-Wallis.

Difficulty strongly influenced performance, with accuracy dropping from 84.7% on very easy questions to 45.4% on difficult ones. Among chatbots that produced citations, Co-Pilot reached 100% existence and relevance.

The authors conclude that chatbots should be supplementary rather than primary learning tools, and that better citation reliability and reasoning could improve future educational use.

What this doesn’t fix

The study is a level IV cross-sectional comparison using only 55 questions from one question bank, so it is a limited test of chatbot performance.

Sources

  1. Peer-reviewedRhode Island medical journal (2013)2026-11-02

The debate