Truace
Subscribe

AI transcription speeds up aphasia speech review but misses most sound errors

Study title: Differential Accuracy of Semantic Versus Phonological Speech Error Transcription in Aphasia: Comparing Large Language Model (Whisper) to Clinician (Speech-Language Pathologist) Performance

Researchers ran 4,535 Philadelphia Naming Test trials from 26 people with poststroke aphasia through OpenAI's Whisper and compared the output with transcripts by an experienced speech-language pathologist. The AI took a few minutes per test where manual work took hours.

Read the study abstract

Purpose: Patient speech samples provide insights into cognitive and linguistic status, yet their utilization has been limited by labor-intensive manual transcription. Advances in automated speech recognition (ASR) with large language models (LLMs) potentially save time, but performance on impaired speech, such as due to poststroke aphasia, remains understudied. Method: We transcribed 4,535 trials of the Philadelphia Naming Test (PNT) administered to 26 people with aphasia (PWA) using OpenAI's Whisper speech-to-t…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 55/100 — Benchmark or lab evaluation · n = 4,535 · 1 independent sourcePeer-reviewedTRV-2026-2343· cite & verify
Differential Accuracy of Semantic Versus Phonological Speech Error Transcription in Aphasia: Comparing Large Language Model (Whisper) to Clinician (Speech-Language Pathologist) Performance
The Role of speech language pathologists in the management of dysphagia by National Center for Health Services Research and Health Care Technology Assessment (U.S.). Public domain
Trace impact readingContested
P 54The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 58The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

The results suggest automation could ease a major bottleneck in aphasia clinical and research work, but only for some patients and error types. Whisper's weak handling of phonemic errors and its rising hallucination rate with severity leave open whether it is safe for broader clinical use.

Main points

  1. The study analysed 4,535 naming-test trials from 26 people with aphasia.
  2. Manual transcription took 180 ± 40 minutes per 175-trial test versus 2.3 ± 0.5 minutes for Whisper.
  3. Whisper's match rate was 88.83% on correct trials, 92.22% on semantic errors and 23.05% on phonemic errors.
  4. Match and hallucination rates correlated with Western Aphasia Battery-Aphasia Quotient scores.
  5. Low-frequency words produced more mismatches than medium- or high-frequency words.

The gain

Whisper transcribed a 175-trial aphasia naming test in about 2.3 minutes, versus about 180 minutes for a speech-language pathologist, and matched clinician transcripts on 92.22% of semantic errors.

The problem

Whisper matched clinician transcripts on only 23.05% of trials with phonemic errors and 75.52% overall, below the over 95% match of trained humans, with more hallucinations in more severe aphasia.

The rundown

Patient speech samples reveal cognitive and linguistic status, but labor-intensive manual transcription has limited their use. The study tested whether a large language model-based speech recognizer could handle impaired speech after stroke.

Whisper's output was compared with gold-standard transcription by an experienced speech-language pathologist. Agreement was analysed by error type, patient severity and word frequency.

The authors conclude that automation is best suited to fluent aphasia types with fewer phonemic distortions, and describe the work as a benchmark of LLM transcription for impaired speech.

What this doesn’t fix

Whisper reliably transcribed semantic but not phonological errors, so automation is best suited to fluent aphasia types with fewer phonemic distortions; the sample was only 26 people with aphasia.

Sources

  1. Peer-reviewedJournal of Speech, Language, and Hearing Research2026-11-02

The debate