Safety of AI speech-to-text transcription of surgical case narratives, with and without LLM error correction

Source article: A Controlled Single-Speaker Evaluation of Ambient Voice Technologies' Speech-to-Text Function Using Narrated Surgical Case-Reports

Ambient voice technologies (AVT) utilise automatic speech recognition (ASR) and generative artificial intelligence for speech-to-text conversion, and are advocated as efficient, accurate tools for clinical documentation. We investigated (n = 8) AVT systems using surgical case-reports as narratives in a controlled single-speaker evaluation prioritising safety outcomes: specifically 3 commercial transcription-systems (Dragon-Medical-One, Heidi-Health, Tortus); 4 ASR speech-to-text Application Programming Interface…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1687· cite & verify

Trace impact readingNegative state

P 73

Both sides are scored from claims and sources, not community votes.

G 68

In brief

Researchers tested eight ambient voice and speech-to-text systems, including commercial clinical products, cloud transcription APIs and an experimental pipeline pairing GPT4oTranscribe with GPT-5 error correction. A single speaker recorded 100 reference transcripts derived from surgical case reports, yielding 800 transcript outputs that were graded for potential patient harm.

The findings matter because these tools are promoted as efficient and accurate for clinical documentation, yet error rates in clinically significant content were high for every system. Language-model correction helped substantially but also created new errors, and the source states that safety and effectiveness in real clinical practice still need evaluation.

Main points

Across 800 outputs, the share containing at least one clinically significant Class 3 error ranged from 30% to 68% depending on the system.
The correction pipeline resolved 51 of 89 significant errors, left 38 in place and introduced 4 new ones.
Of 53 errors with potential for severe harm or death, 39.6% involved medication type or dose and 30.2% involved investigations or laboratory results.
Longer reference transcripts raised the odds of Class 3 errors for four systems, with odds ratios of 2.65-3.34 per additional 100 words.
On medical terminology, GPT4oTranscribe-Corrected-5 (DWER 3.60%) and Heidi-Health (5.67%) did best, while Amazon-Medical-Transcribe was worst at 24.03%.

The gain

Adding GPT-5 generative error correction to a GPT4oTranscribe pipeline reduced the share of transcripts with clinically significant Class 3 errors from 53% to 30%, resolving 51 of 89 such errors.

The problem

Every tested ambient voice system produced clinically significant Class 3 errors in 30-68% of transcripts, including 53 errors with potential for severe harm or death, mostly in medication and laboratory result details.

The rundown

The study compared three commercial transcription systems (Dragon-Medical-One, Heidi-Health, Tortus), four speech-to-text APIs (Speechmatics-Enhanced, Amazon-Medical-Transcribe, Whisper, GPT4oTranscribe) and one experimental two-stage pipeline. The reference transcripts totalled 32,897 words, averaging 329 words each.

The primary outcome was the proportion of outputs with at least one Class 3 error graded for potential harm. Secondary outcomes were domain word error rate against SNOMED-CT, lexical accuracy via ROUGE, and semantic similarity via BERT and BART scores. Performance differed significantly across systems on these transcription metrics (P < 0.001).

The pipeline comparison classified errors in the raw ASR output as resolved, remaining or newly introduced. Newly introduced errors affected 3% of transcript outputs, so the correction layer reduced but did not remove clinically significant risk.

What this doesn’t fix

The evaluation used a single speaker and narratives derived from surgical case reports in a controlled setting, and the authors state that safety and effectiveness in clinical practice require further evaluation.