LLM-assisted CPT coding from pediatric orthopaedic operative notes

Source article: Artificial Intelligence-Assisted Current Procedural Terminology Coding in Pediatric Orthopaedic Surgery: Promise or Premature?

Background: Accurate assignment of Current Procedural Terminology (CPT) codes is essential for reimbursement, compliance, and quality reporting in pediatric orthopaedic surgery. Manual coding remains resource-intensive and prone to error and variability. Large language models (LLMs) represent a potential solution for automating CPT assignment directly from operative documentation. This study evaluated the performance of ChatGPT-4o and ChatGPT-5 in generating accurate CPT codes from deidentified pediatric orthopa…

By the Truvace deskOctober 16, 20262 min read1 sourcePeer-reviewedTRV-2026-1622· cite & verify

Artificial Intelligence-Assisted Current Procedural Terminology Coding in Pediatric Orthopaedic Surgery: Promise or Premature? — Occupational therapy posters (Reeve 041502), National Museum of Health and Medicine (3298009001) by National Museum of Health and Medicine. CC BY 2.0 · https://creativecommons.org/licenses/by/2.0

Trace impact readingContested

P 69

Both sides are scored from claims and sources, not community votes.

G 65

In brief

Researchers tested two versions of ChatGPT on turning deidentified pediatric orthopaedic operative notes into billing codes. Using a standard zero-shot prompt on 20 notes from 7 surgeons, they compared model output with codes assigned by the institutional billing department. The newer model clearly outperformed the older one.

The results suggest language models could cut some of the administrative load of coding, especially for simple single-code cases. But the small sample, retrospective design, and remaining error rates mean the tools still need human review, and larger multi-institution validation has yet to be done.

Main points

ChatGPT-5 reached an overall F1 score of 0.69 versus 0.48 for ChatGPT-4o, with precision 0.67 versus 0.45 and recall 0.74 versus 0.56.
Both models did better on single-code cases; ChatGPT-5 scored 0.76 on those versus 0.61 on multicode cases.
ChatGPT-4o overcoded in 70% of cases and undercoded in 60%, while ChatGPT-5 overcoded in 25% and undercoded in 30%.
The authors call for larger validation, domain-specific fine-tuning, and electronic health record integration before clinical billing use.

The gain

On 20 pediatric orthopaedic operative notes, ChatGPT-5 raised exact-match CPT coding accuracy to 60% from 30% for ChatGPT-4o, supporting use as a supervised first-pass coding aid.

The problem

Even the better model overcoded 25% and undercoded 30% of cases, leaving error rates above acceptable thresholds for unsupervised billing deployment.

The rundown

The study was a retrospective observational comparison using notes collected between January and April 2025. Notes were categorized as single-code or multicode, and outputs were adjudicated as true positives, false positives, or false negatives against billing department codes.

Accurate CPT assignment matters for reimbursement, compliance, and quality reporting, while manual coding is resource-intensive and variable. The authors frame the models as augmentative tools rather than replacements for human coders.

The paper rates its evidence as Level III, a retrospective comparative study. Its conclusion is that single-code pediatric orthopaedic cases may be suitable earlier for supervised use.

What this doesn’t fix

The study used only 20 notes from one institution's case mix, and the authors state that larger-scale validation and integration studies are needed before broader implementation.