Truace
Subscribe

AI chatbots classified knee X-rays correctly only about half the time, well below surgeons

Study title: Comparative accuracy of large language models and orthopaedic clinicians in knee osteoarthritis and tibial plateau fracture classification systems

Researchers gave 100 Radiopaedia knee cases, each with anteroposterior and lateral radiographs, to five orthopaedic surgeons, five residents and four multimodal language models. Each case was classified under one of five systems, and three senior surgeons set the reference standard.

Read the study abstract

Background: Multimodal large language models (LLMs) can process radiographic images, but their accuracy across orthopaedic classification systems requires further evaluation. This study compared the classification accuracy of four LLMs with that of orthopaedic surgeons and residents. Methods: This cross-sectional study included 100 cases obtained from Radiopaedia, with 20 cases each for the Kellgren-Lawrence, Ahlback, Schatzker, Hohl and Moore, and AO Foundation/Orthopaedic Trauma Association (AO/OTA) classifica…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-2253· cite & verify
Peer-reviewed study: The Knee

In brief

The gap suggests these general-purpose models are not yet dependable for radiographic classification in this setting, though the test used a narrow educational dataset. Whether results hold with other prompts, languages, larger or more representative case sets, or in clinical practice remains untested.

Main points

  1. Surgeons reached 91.2% accuracy, residents 70.2%, and the LLMs combined 51.0%.
  2. LLM accuracy was about 70% for Kellgren-Lawrence, 68.8% for Ahlback and 67.5% for AO/OTA, but 28.8% for Hohl and Moore and 20.0% for Schatzker.
  3. Model accuracy ranged from 58.0% for Claude Opus 4.1 and 57.0% for GPT-5.5 to 45.0% for Gemini 2.5 Pro and 44.0% for DeepSeek-VL2.
  4. Surgeons had the highest observed accuracy across all five classification systems.

The problem

Four multimodal LLMs classified knee radiographs correctly only about half the time, well below surgeons and residents, with especially weak performance on Schatzker and Hohl and Moore fracture systems.

The rundown

The study was cross-sectional and used 20 cases for each of Kellgren-Lawrence, Ahlback, Schatzker, Hohl and Moore, and AO/OTA classifications, producing 1400 classification assessments in total.

AI evaluations took place in April 2026 using a standardized Turkish-language prompt with no classification definitions or examples, and only each model's first response was recorded.

Because paired case-level classifications were unavailable, the analyses were descriptive. The authors state the findings do not establish educational or clinical utility.

Sources

  1. Peer-reviewedThe Knee2026-10-24

The debate