Moderate evidence
More
Newer AI model misread hip surgery notes far less often, study of 1,000 notes finds
Study title: New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction
- Evidence type
- Controlled or validated comparison
- Sample
- n = 1,000
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2961cite or check it
Study title: New Large Language Models Outperform Older Models and Hallucinate Less in Operative Note Data Extraction
Researchers compared two locally hosted large language models on pulling three data points from 1,000 primary total hip arthroplasty operative notes written by 47 surgeons. The model released in 2026 beat the 2024 model on every task: surgical approach 98% versus 96%, fixation technique 98% versus 96% and bearing surface 93% versus 89%.
Read the study abstract
Background: Recent improvements in large language models (LLMs) may provide a scalable solution for registry data curation. However, trust in model outputs remains limited by hallucinations, wherein models invent information. Given the rapid pace of LLM development, the purpose of this study was to compare the accuracy and hallucination rates of a "newer" and an "older" and previously evaluated LLM in extracting total hip arthroplasty (THA) operative variables. Methods: Surgical approach, fixation technique, and…
The gain
In a study of 1,000 total hip arthroplasty operative notes, a 2026 language model extracted surgical approach, fixation and bearing surface at 98%, 98% and 93% accuracy versus 96%, 96% and 89% for a 2024 model, and hallucinated far less.
In brief
The bigger gap was in hallucinations, where models invent information. In 100 manually reviewed output reasonings per model, the newer model hallucinated 5%, 3% and 11% of the time for approach, fixation and bearing surface, against 49%, 20% and 49% for the older model.
The work matters for registry data curation, where trust in model output is limited by invented content. The authors conclude the models are getting better at following directions, but remaining hallucinations still limit how far the output can be trusted.
Main points
- Surgical approach, fixation technique and bearing surface were manually abstracted from 1,000 primary total hip arthroplasty operative notes by 47 surgeons.
- Prompts were developed on 25% of the notes and tested on the remaining 75%.
- Accuracy for the newer model was 98%, 98% and 93%, versus 96%, 96% and 89% for the older model.
- Hallucination rates were 5%, 3% and 11% for the newer model versus 49%, 20% and 49% for the older one.
- The best-performing prompts for the newer model were simpler than those for the older model.
- Hallucinations were assessed manually on 100 sample output reasonings from each model.
The full story
The team abstracted three variables by hand from 1,000 operative notes from 47 surgeons. They used 25% of the notes to iteratively develop prompts for two locally hosted models, one released in 2024 and one in 2026, then tested the pipeline on the other 75%.
The newer model scored higher on all three tasks, with the largest gain on bearing surface (93% versus 89%). For each task its best prompt was simpler than the best prompt for the older model.
For hallucinations, reviewers manually assessed 100 sample reasonings from each model. The newer model's rates were significantly lower across all domains, for example 11% versus 49% for bearing surface. The authors say lower rates persist, which limits trustworthiness.
Sources
1 on file- The Journal of Arthroplastydoi:10.1016/j.arth.2026.11.047
Talk it over
The newer model hallucinated 5%, 3% and 11% of the time, against 49%, 20% and 49% for the older one. How would you weigh that improvement against the hallucinations that remain?
Newer AI model misread hip surgery notes far less often, study of 1,000 notes findshttps://truvace.com/item/new-large-language-models-outperform-older-models-and-hallucinate-less-in-operat
The newer model hallucinated 5%, 3% and 11% of the time, against 49%, 20% and 49% for the older one. How would you weigh that improvement against the hallucinations that remain?
Answer in the threadReviewers manually assessed 100 sample reasonings from each model, and the notes came from 47 surgeons. What can that design tell us about how the models would handle notes from other surgeons or hospitals?
Answer in the threadThe authors say remaining hallucinations limit how far the output can be trusted for registry data. What result would make you comfortable letting a model fill a hip surgery registry without a person checking?
Answer in the thread
If you have more time
Look up how registries currently collect details like fixation and bearing surface. Who does that work now, and what would change if a model did it?
Answer in the threadThe best prompts for the newer model were simpler than those for the older model. What would you ask the researchers about how much prompt tuning on 25% of the notes shaped the results?
Answer in the thread
ace


What readers say