Truace
Subscribe

LLM extraction of adverse drug events from clinical notes

Source article: Relationship extraction for adverse drug events in clinical notes using large language models: a retrospective cross-sectional study

Background: Adverse drug events (ADEs) are a critical indicator of patient safety but are often documented only in free-text clinical notes. The potential of recent advances in natural language processing (NLP), particularly generative large language models (LLMs), to identify ADEs remains understudied. This study aimed to compare the performance of multiple LLMs in identifying ADE-Drug relationships in inpatient and ambulatory clinical notes. Methods: We used clinical notes from the 2018 National NLP Clinical C…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1807· cite & verify
Relationship extraction for adverse drug events in clinical notes using large language models: a retrospective cross-sectional study
Emoji Use in the Electronic Health Record by David A. Hanauer, Gavin C. Raab, Shira N. Hanauer, Lisa Ferguson, Kellen McClain, Guan Wang, Michelle Rozwadowski, Sung W. Choi. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0
Trace impact readingContested
P 67The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers compared GPT-4o, GPT-4o-mini and Llama 3.2-3B, along with fine-tuned variants, on identifying links between adverse drug events and drugs in free-text notes. The data came from the 2018 n2c2 inpatient dataset and outpatient encounters at a large New England academic medical center. GPT-4o and GPT-4o-mini performed best, and GPT-4o led on F1 score.

Adverse drug events are an important patient-safety indicator, yet they are often recorded only in free text, so automated extraction could support surveillance. The reported F1 scores are moderate, and the errors seen in manual review show that clinical use would require care. The study frames its results as a foundation, not proof of deployed safety gains.

Main points

  1. Of 1048 notes in the local test split, 450 (42.9%) contained at least one adverse drug event.
  2. The n2c2 test set had 202 notes, and 155 (76.7%) contained at least one adverse drug event.
  3. GPT-4o reached an F1 of 0.524 versus 0.381 for GPT-4o-mini, with a more balanced precision-recall profile.
  4. Manual review of the two best models used categories such as exact semantic match, partial match, missing ADE and wrong.

The gain

GPT-4o identified adverse drug event and drug relationships in inpatient and outpatient notes better than GPT-4o-mini, suggesting a basis for LLM-assisted drug safety surveillance.

The problem

Both models made characteristic errors, including misclassifying pre-existing allergies and conflating therapeutic indications with adverse effects, and overall F1 scores stayed modest.

The rundown

The study was a retrospective cross-sectional analysis using 505 inpatient notes from the n2c2 dataset and 2555 outpatient notes from October 2018 to December 2019. Notes were split into snippets before being given to the models.

Models tested were GPT-4o, GPT-4o-mini, Llama 3.2-3B and instruction fine-tuned variants, including low-rank adapters for Llama. Performance was scored with strict and relaxed precision, recall and F1.

The two best models were then manually assessed. The authors report that both captured adverse drug events in explicit and complex contexts, but they also misclassified pre-existing allergies and sometimes confused indications with adverse effects.

What this doesn’t fix

Reported F1 scores were modest, and both top models misclassified pre-existing allergies and sometimes confused therapeutic indications with adverse effects.

Sources

  1. Peer-reviewedeBioMedicine2026-10-17

The debate