63/100

Moderate evidence

Cite
More
EmailLinkedInX

AI reading admission notes predicted stroke recovery better than risk scores, study finds

Study title: Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke

Evidence type
Observational study
Sample
n = 9,485
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke

Researchers at NYU Langone tested whether language models can estimate how well stroke patients will recover using only the history and physical notes written at admission. After fine-tuning, Llama-3.1-8B reached an AUROC of 0.87 for 90-day outcome and 0.84 for outcome at discharge.

Read the study abstract

Background and objectives: Accurate prediction of functional outcomes after acute ischemic stroke can guide clinical decision-making and resource allocation. Prior work has relied predominantly on structured predictors and conventional machine learning, while the ability of large language models (LLMs) to infer modified Rankin Scale (mRS) scores from routine clinical admission notes is largely unexplored. Methods: We evaluated encoder (Bidirectional Encoder Representations from Transformers [BERT], NYUTron) and…

Large Language Models Predict Functional Outcomes After Acute Ischemic Stroke
"Telemedicine Consult" by Intel Free Press, CC BY-SA 2.0.

The gain

In a retrospective NYU Langone analysis of 1,898 stroke patients' admission notes, a fine-tuned Llama model predicted 90-day functional outcome with an AUROC of 0.87, beating four published baselines that scored 0.79 to 0.82.

In brief

For 90-day prediction, the fine-tuned model beat structured, hybrid and clinical risk score baselines, which scored between 0.79 and 0.82 (all P < .05). The authors say such text-based tools could fit into clinical workflows without manual data extraction.

The work is a retrospective test on one hospital system's registry data, with the most recent 12 months held out. Differences between model types varied by metric and cannot be pinned on architecture alone, according to the authors.

Main points

  1. The study used 9,485 admission notes for discharge prediction and 1,898 notes for 90-day prediction from the NYU Langone Get With The Guidelines-Stroke registry, 2016 to 2025.
  2. Models tested were BERT, NYUTron, Llama-3.1-8B and MedGemma-4B, each in frozen and fine-tuned settings, plus ModernBERT-base as a long-context check.
  3. Fine-tuned Llama reached AUROC 0.84 for discharge and 0.87 for 90-day outcome, with precision-recall areas of 0.81 and 0.84.
  4. Baselines for 90-day outcome scored AUROC 0.82 (Zhang et al), 0.82 (Sung et al), 0.80 (THRIVE) and 0.79 (SPAN-100).
  5. Before fine-tuning, the clinically pretrained NYUTron encoder outperformed all other models.
  6. Binary accuracy was similar among NYUTron, Llama and MedGemma, and ModernBERT-base did not beat NYUTron.

The full story

The team framed the task as a binary call on modified Rankin Scale scores: good outcome (0 to 2) versus poor outcome (3 to 6). They compared encoder models (BERT, NYUTron) and generative models (Llama-3.1-8B, MedGemma-4B), and the most recent 12 months of data were held out for testing.

Without fine-tuning, NYUTron, which was pretrained on clinical text, led every other model. Fine-tuning changed the ranking: Llama had the highest discrimination, though NYUTron, Llama and MedGemma had similar binary accuracy.

A long-context ModernBERT-base analysis did not beat NYUTron or remove differences in predicting the exact mRS score. The authors conclude that differences between model classes varied by metric and cannot be attributed to architecture alone.

Sources

1 on file
  1. Neurosurgerydoi:10.1227/neu.0000000000004298
    Peer-reviewed2026-11-19

Talk it over

Four questions to start on
Who gets this benefit, and who was left out of the study or story?

Start with a question

AI reading admission notes predicted stroke recovery better than risk scores, study findshttps://truvace.com/item/large-language-models-predict-functional-outcomes-after-acute-ischemic-stroke

  1. Who gets this benefit, and who was left out of the study or story?

    The gain In a retrospective NYU Langone analysis of 1,898 stroke patients' admission notes, a fine-tuned Llama model predicted 90-day functional outcome with an AUROC of 0.87, beating four published baselines that scored 0.79 to 0.82.

    Answer in the thread
  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on an observational study with a sample of 9,485, from Neurosurgery.

    Answer in the thread
  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The models were built and tested retrospectively on registry notes from a single health system, and differences between model classes varied by metric, so they cannot be attributed to architecture alone.

    Answer in the thread
  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say