Truace
Subscribe

A Hybrid Human-AI Workflow for Ultrasound Report Quality Control: A Multicenter Performance and Efficiency Evaluation

Rationale and objectives: This study evaluated a hybrid human-AI workflow for ultrasound report quality assurance using multicenter retrospective data. Materials and methods: 786 anonymized ultrasound reports covering 22 examination categories were collected from four hospitals. Reports containing naturally occurring or implanted errors were matched across arms. GPT-4o and DeepSeek-v3 performed initial screening before review by six human reviewers (HRs). HR with blinded expert over-reading (HR-ex) served as con…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1805· cite & verify
A Hybrid Human-AI Workflow for Ultrasound Report Quality Control: A Multicenter Performance and Efficiency Evaluation
Annual report - National Cancer Institute (U.S.), Division of Resources, Centers and Community Activities (FY 1981, Part 4-A) (IA annualreportnati19814nati) by Bethesda, Md. : National Cancer Institute. Public domain

In brief

Researchers tested a workflow in which large language models screened ultrasound reports for errors before human reviewers checked them. The test used 786 anonymized reports from four hospitals across 22 examination categories, with some errors implanted. A second validation used 1680 consecutive reports with no implanted errors.

The result suggests AI-assisted screening could speed up report auditing without lowering accuracy below that of a single physician. However, it did not match expert over-reading and let more errors through. The authors say prospective real-world validation is needed before routine clinical use.

Main points

  1. Accuracy was 80.2% for GPT-4o hybrid review, 82.5% for DeepSeek-v3 hybrid review and 82.3% for human review, versus 87.3% for expert over-reading.
  2. Human verification reduced both false negatives and false positives produced by the language models.
  3. Severity assessment did not indicate preferential omission of clinically significant errors.
  4. In the supplementary set with 9.6% error prevalence, the DeepSeek hybrid had a 22.6% false-negative rate versus 24.3% for human review.

The gain

Pairing GPT-4o or DeepSeek-v3 screening with human review matched single-reviewer accuracy on ultrasound reports while cutting median audit time from 51.0 s to about 11 to 12 s.

The rundown

The study used retrospective data in which reports with naturally occurring or implanted errors were matched across arms. GPT-4o and DeepSeek-v3 did the first screening, and six human reviewers then examined the results. Human review with blinded expert over-reading served as the control.

On review time, the hybrid arms took a median of 11.0 and 12.0 seconds per report, compared with 51.0 seconds for expert over-reading. In the supplementary validation, median times were 15.4 seconds for the DeepSeek-assisted hybrid and 24.3 seconds for human review.

In that validation, false-positive rates were 7.7% for the hybrid and 10.0% for human review. Both approaches still missed more than a fifth of errors, at 22.6% and 24.3% respectively.

Sources

  1. Peer-reviewedAcademic Radiology2026-10-17

The debate