70/100

Moderate evidence

Cite
More
EmailLinkedInX

Newer AI model misread hip surgery notes far less often, study of 1,000 notes finds

Study title: New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction

Evidence type
Controlled or validated comparison
Sample
n = 1,000
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction

Researchers compared two locally hosted large language models on pulling three data points from 1,000 primary total hip arthroplasty operative notes written by 47 surgeons. The model released in 2026 beat the 2024 model on every task: surgical approach 98% versus 96%, fixation technique 98% versus 96% and bearing surface 93% versus 89%.

Read the study abstract

Background: Recent improvements in large language models (LLMs) may provide a scalable solution for registry data curation. However, trust in model outputs remains limited by hallucinations, wherein models invent information. Given the rapid pace of LLM development, the purpose of this study was to compare the accuracy and hallucination rates of a "newer" and an "older" and previously evaluated LLM in extracting total hip arthroplasty (THA) operative variables. Methods: Surgical approach, fixation technique, and…

The gain

In a study of 1,000 total hip arthroplasty operative notes, a 2026 language model extracted surgical approach, fixation and bearing surface at 98%, 98% and 93% accuracy versus 96%, 96% and 89% for a 2024 model, and hallucinated far less.

In brief

The bigger gap was in hallucinations, where models invent information. In 100 manually reviewed output reasonings per model, the newer model hallucinated 5%, 3% and 11% of the time for approach, fixation and bearing surface, against 49%, 20% and 49% for the older model.

The work matters for registry data curation, where trust in model output is limited by invented content. The authors conclude the models are getting better at following directions, but remaining hallucinations still limit how far the output can be trusted.

Main points

  1. Surgical approach, fixation technique and bearing surface were manually abstracted from 1,000 primary total hip arthroplasty operative notes by 47 surgeons.
  2. Prompts were developed on 25% of the notes and tested on the remaining 75%.
  3. Accuracy for the newer model was 98%, 98% and 93%, versus 96%, 96% and 89% for the older model.
  4. Hallucination rates were 5%, 3% and 11% for the newer model versus 49%, 20% and 49% for the older one.
  5. The best-performing prompts for the newer model were simpler than those for the older model.
  6. Hallucinations were assessed manually on 100 sample output reasonings from each model.

The full story

The team abstracted three variables by hand from 1,000 operative notes from 47 surgeons. They used 25% of the notes to iteratively develop prompts for two locally hosted models, one released in 2024 and one in 2026, then tested the pipeline on the other 75%.

The newer model scored higher on all three tasks, with the largest gain on bearing surface (93% versus 89%). For each task its best prompt was simpler than the best prompt for the older model.

For hallucinations, reviewers manually assessed 100 sample reasonings from each model. The newer model's rates were significantly lower across all domains, for example 11% versus 49% for bearing surface. The authors say lower rates persist, which limits trustworthiness.

Sources

1 on file
  1. The Journal of Arthroplastydoi:10.1016/j.arth.2026.11.047
    Peer-reviewed2026-11-12

Talk it over

Three questions to start on, two more if you have time
The newer model hallucinated 5%, 3% and 11% of the time, against 49%, 20% and 49% for the older one. How would you weigh that improvement against the hallucinations that remain?

Start with a question

Newer AI model misread hip surgery notes far less often, study of 1,000 notes findshttps://truvace.com/item/new-large-language-models-outperform-older-models-and-hallucinate-less-in-operat

  1. The newer model hallucinated 5%, 3% and 11% of the time, against 49%, 20% and 49% for the older one. How would you weigh that improvement against the hallucinations that remain?

    Answer in the thread
  2. Reviewers manually assessed 100 sample reasonings from each model, and the notes came from 47 surgeons. What can that design tell us about how the models would handle notes from other surgeons or hospitals?

    Answer in the thread
  3. The authors say remaining hallucinations limit how far the output can be trusted for registry data. What result would make you comfortable letting a model fill a hip surgery registry without a person checking?

    Answer in the thread
If you have more time
  • Look up how registries currently collect details like fixation and bearing surface. Who does that work now, and what would change if a model did it?

    Answer in the thread
  • The best prompts for the newer model were simpler than those for the older model. What would you ask the researchers about how much prompt tuning on 25% of the notes shaped the results?

    Answer in the thread

What readers say