Truace
Subscribe

AI-generated literature reviews on shoulder and elbow surgery topics

Source article: A focused PubMed artificial intelligence search program creates better literature reviews than general purpose artificial intelligence and is comparable to trainees in shoulder and elbow surgery

Background: As the use of artificial intelligence (AI) expands into orthopedic surgery, the potential to use AI for review and summation of literature leaves researchers and clinicians questioning the quality of AI-driven literature reviews. This study aims to compare AI-generated literature reviews to those done by orthopedic surgery trainees. Methods: ChatGPT-4o and The-Literature, a directed AI platform, were asked to produce a literature review on 5 shoulder and elbow-related research topics. The primary out…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1830· cite & verify
A focused PubMed artificial intelligence search program creates better literature reviews than general purpose artificial intelligence and is comparable to trainees in shoulder and elbow surgery
Polio physical therapy by Photo Credit: Content Providers: CDC/Charles Farmer. Public domain
Trace impact readingContested
P 63The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 64The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers asked ChatGPT-4o and a directed AI platform called The-Literature to write literature reviews on five shoulder and elbow topics. Their output was compared with reviews by a medical student, a resident, and a fellow. Two fellowship-trained shoulder and elbow surgeons graded the blinded reviews on a 1-5 scale.

The results suggest that how an AI tool is built and scoped may matter for reliability in clinical research tasks. A restricted search-based program performed at trainee level, while an open chatbot did poorly because it invented sources. The study covers only five topics and two expert graders, so it is unclear how the findings extend to other fields or newer models.

Main points

  1. Average overall ratings were 2.90 for the medical student, 3.90 for the resident, 3.50 for the fellow, 2.30 for ChatGPT-4, and 3.40 for The-Literature.
  2. The resident and fellow reviews were significantly better than ChatGPT-4, but the medical student review was not.
  3. The-Literature was rated significantly better than ChatGPT-4 and no trainee review significantly outperformed it.
  4. 58.8% of ChatGPT-4's references were fabricated, which the authors identify as the main source of its poor utility.

The gain

A directed AI search platform produced shoulder and elbow literature reviews rated comparably to orthopedic trainees and better than a general-purpose chatbot.

The problem

A general-purpose large language model produced weaker reviews, with most of its cited references fabricated.

The rundown

The study design used five shoulder and elbow research topics. Outputs from ChatGPT-4, The-Literature, a fourth-year medical student, a third-year orthopedic resident, and a sports medicine fellow were blinded before grading.

Graders scored evidence and analysis, conclusion, references, and overall quality. Overall ratings had notable variability, with a standard deviation of 1.16 for ChatGPT-4 compared with 0.53 for the fellow.

The authors conclude that a limited, directed AI search program was comparable to orthopedic surgery trainees and outperformed an open large language model, which had poor overall utility.

What this doesn’t fix

The comparison rests on five topics, a small number of human reviewers, and two graders, and tested ChatGPT-4 rather than other models, which limits generalization.

Sources

  1. Peer-reviewedJSES International2026-09-04

The debate