Moderate evidence
More
AI sorted stroke registry notes and found 1,345 more complications than the structured fields recorded
Study title: Decoding Complications: Large Language Model-Based Classification of Free-Text Entries in Clinical Stroke Registries
- Evidence type
- Controlled or validated comparison
- Sample
- n = 9,146
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2994cite or check it
Study title: Decoding Complications: Large Language Model-Based Classification of Free-Text Entries in Clinical Stroke Registries
Researchers built a workflow using a large language model to classify free-text adverse-event entries in the German Stroke Registry. They started with 11,216 entries from 18,069 patients and kept 9,146 informative ones. An LLM-guided taxonomy produced 89 leaf labels, while embedding-based clustering separated the entries poorly.
Read the study abstract
Clinical registries capture adverse events (AEs) in free-text fields that are rarely analyzed. We developed a reproducible large language model (LLM)-assisted workflow using 11,216 free-text entries from 18,069 patients in the German Stroke Registry; 9,146 informative entries were retained after filtering. Embedding-based clustering showed poor separation, whereas an LLM-guided taxonomy identified 89 leaf labels. On the model-assisted 1,000-entry development set used for model and prompt selection, GPT-5.4 reach…

The gain
In a German Stroke Registry study, GPT-5.4 labelled 300 held-out free-text entries at 96.5% weighted F1 against clinician consensus, and the free text documented 1,345 more adverse-event instances (+19.6%) than the structured field for comparable concepts.
In brief
On 300 held-out entries, frozen GPT-5.4 labels reached a weighted F1 of 96.5% against clinician consensus. A locally run open model, Qwen 3.5 27B, reached 90.5%, which suggests a route for sites that cannot send data to external services.
The free text recorded 1,345 more instances of adverse events than the structured field where a comparable one existed, and 7,546 instances with no structured equivalent, most often pneumonia, urinary tract infection and delirium. Adding these complication concepts improved prediction of good functional outcome and death only slightly, and the improvement was minimal once the discharge NIHSS score was included.
Main points
- The registry data covered 11,216 free-text entries from 18,069 patients; 9,146 informative entries remained after filtering.
- Embedding-based clustering separated entries poorly, so the team used an LLM-guided taxonomy with 89 leaf labels.
- On a 1,000-entry development set used for model and prompt selection, GPT-5.4 reached F1 92.4% (95% CI 91.4-93.4).
- On 300 held-out entries, weighted F1 against clinician consensus was 96.5% for GPT-5.4 and 90.5% for locally run Qwen 3.5 27B.
- Free text documented 1,345 extra instances (+19.6%) over 6,867 structured instances, plus 7,546 instances of events with no structured equivalent.
- Complication concepts raised the area under the curve by 0.025 for good functional outcome and 0.018 for death, with minimal gains after adding the discharge NIHSS score.
- On 1,979 public coronary-trial fragments from ClinicalTrials.gov, the software ran end to end.
The full story
The team addressed a common gap: registries capture adverse events in free-text fields that are rarely analysed. They filtered 11,216 entries down to 9,146 informative ones and tried embedding-based clustering, which separated them poorly. An LLM-guided taxonomy then produced 89 leaf labels.
The 1,000-entry development set was model-assisted and used to pick the model and prompt, where GPT-5.4 reached F1 92.4%. The more telling test was 300 held-out entries compared with clinician consensus, where frozen GPT-5.4 labels scored 96.5% weighted F1 and local Qwen 3.5 27B scored 90.5%.
Compared with structured documentation, the free text recorded 19.6% more instances of comparable adverse-event concepts and a further 7,546 instances without a structured equivalent. Adding the complication concepts raised AUC by 0.025 for good functional outcome and 0.018 for death beyond baseline covariates and the structured count, but gains were minimal after including the discharge NIHSS score. A run on 1,979 coronary-trial fragments showed the software works on a second text corpus.
Sources
1 on file- Journal of Medical Systemsdoi:10.1007/s10916-026-02486-y
Talk it over
The free text recorded 1,345 more adverse-event instances than the structured field, yet complication concepts added only 0.025 to the prediction of good functional outcome. How do you weigh the extra detail against that small predictive gain?
AI sorted stroke registry notes and found 1,345 more complications than the structured fields recordedhttps://truvace.com/item/decoding-complications-large-language-model-based-classification-of-free-text-en
The free text recorded 1,345 more adverse-event instances than the structured field, yet complication concepts added only 0.025 to the prediction of good functional outcome. How do you weigh the extra detail against that small predictive gain?
Answer in the threadGPT-5.4 scored 96.5% weighted F1 on 300 held-out entries, while the 1,000-entry development set was model-assisted. What does that difference in how the two sets were checked tell you about how far to trust each number?
Answer in the threadA locally run open model, Qwen 3.5 27B, reached 90.5% against 96.5% for GPT-5.4. If you ran a stroke registry site that cannot send data to external services, which model would you choose and why?
Answer in the thread
If you have more time
Pneumonia, urinary tract infection and delirium were the most common events with no structured equivalent. Who in a hospital you know would use that information, and for what?
Answer in the threadThe software ran end to end on 1,979 coronary-trial fragments from ClinicalTrials.gov. What evidence would show it also classifies those entries accurately?
Answer in the thread
ace
What readers say