Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2294
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-28T06:46:17.963925Z
status: published
lens: trace
sector: education
headline: Performance of AI Chatbots in Adult Reconstruction Education: A Comparative Evaluation Using AAOS ResStudy Questions
dek: Background: Artificial intelligence (AI) chatbots are increasingly incorporated into orthopedic education, yet their ability to handle complex, clinically oriented questions remains unclear. This study evaluates the accuracy, response time, and citation reliability of contemporary AI chatbots when answering adult reconstruction questions from the AAOS ResStudy question bank.
Methods: Fifty-five multiple-choice adult reconstruction questions were extracted and categorized by anatomic region (hip vs. knee), concep…
gain_title: GPT-4o answered 80% of adult reconstruction questions correctly, above the 75% score of AAOS members, and Co-Pilot's citations were all real and relevant.
problem_title: Overall chatbot accuracy was only 67.5%, falling to 45.4% on difficult questions and 63.1% on prognosis, so chatbots struggle with applied reasoning.
trace_subject: AI chatbot accuracy on AAOS ResStudy adult reconstruction questions
gain_reading: GPT-4o answered 80% of adult reconstruction questions correctly, above the 75% score of AAOS members, and Co-Pilot's citations were all real and relevant.
gain_evidence: GPT-4o performed best (80%), outperforming AAOS members (75%) | Co-Pilot achieved 100% citation existence and relevance
problem_reading: Overall chatbot accuracy was only 67.5%, falling to 45.4% on difficult questions and 63.1% on prognosis, so chatbots struggle with applied reasoning.
problem_evidence: Overall chatbot accuracy was 67.5% | accuracy dropping from 84.7% on very easy questions to 45.4% on difficult ones | they continue to struggle with applied reasoning tasks such as prognosis and management
quick_read: Researchers gave 55 multiple-choice adult reconstruction questions from the AAOS ResStudy bank to seven chatbots, including several ChatGPT versions, Gemini, Claude and Co-Pilot. They recorded accuracy, response time and whether the cited sources existed and were relevant.

The results suggest chatbots can help with descriptive orthopedic knowledge but are not reliable enough to serve as a primary study resource. Performance fell sharply as questions got harder, and the study tested only one question bank, so how this carries over to real training or clinical decisions is unclear.
limitation: The study is a level IV cross-sectional comparison using only 55 questions from one question bank, so it is a limited test of chatbot performance.
tag: Dual reading
key_points: Overall accuracy across the seven chatbots was 67.5%. | GPT-4o scored 80%, ahead of AAOS members at 75%; Co-Pilot ranked second at 70.9%. | Accuracy was highest on clinical presentation (78.6%) and lowest on prognosis (63.1%). | Average response time was 3.6 seconds, with no significant correlation between speed and accuracy.
rundown: The 55 questions were categorized by anatomic region (hip or knee), concept (clinical presentation, prognosis, management) and AAOS-defined difficulty. Statistical tests included chi-square, t-tests, Pearson correlation, ANOVA and Kruskal-Wallis.

Difficulty strongly influenced performance, with accuracy dropping from 84.7% on very easy questions to 45.4% on difficult ones. Among chatbots that produced citations, Co-Pilot reached 100% existence and relevance.

The authors conclude that chatbots should be supplementary rather than primary learning tools, and that better citation reliability and reasoning could improve future educational use.
sources:
- peer_reviewed | Rhode Island medical journal (2013) | https://pubmed.ncbi.nlm.nih.gov/42945521/ | 2026-11-02
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

59c26e12bc4f289dfbe75f23bb696a8e78954f2595497cb208cdb489dfdded7b

previous

0000000000000000000000000000000000000000000000000000000000000000