A language model framework for sequence modeling of audit logs to characterize clinician-electronic health record interactions

Objective: Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (eg, total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions. Materials and methods: Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (…

By the Truvace deskOctober 15, 20262 min read1 sourcePeer-reviewedTRV-2026-1423· cite & verify

A language model framework for sequence modeling of audit logs to characterize clinician-electronic health record interactions — "NextGen is unusable" by juhansonin is licensed under CC BY 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/2.0/.

In brief

Researchers used more than 295 million inpatient audit log actions recorded between 2019 and 2024 to fine-tune Llama-3-8B to predict a clinician's next action in the electronic health record. They compared two encodings, symbolic field-based tokens and word-based natural-language descriptions, against a first-order Markov baseline.

The word-based model was the most accurate and held up better than the baseline on later out-of-sample periods, though its accuracy fell over time. The findings point toward using natural-language LLMs to study clinical workflows and safety, but the work reports only next-action prediction accuracy rather than effects on care or on clinicians.

Main points

Training data comprised more than 295 million EHR audit log actions from inpatient settings, 2019 to 2024.
In the early-period test set, the word-based LLM scored 0.7473 accuracy versus 0.3464 for the field-based LLM and 0.4230 for the Markov baseline.
Out-of-sample accuracy for the word-based LLM was 0.6555 in OOS-1 and 0.6286 in OOS-2, versus 0.3139 and 0.2720 for the Markov baseline.
The authors suggest longer sequential context and word-based action descriptions may improve audit log sequence modeling.

The problem

Next-action prediction accuracy of the word-based model declined in later out-of-sample periods, indicating temporal drift in the learned workflow patterns.

The rundown

Most audit log research has relied on aggregated measures such as total time spent in the EHR. This study instead modeled the order of actions, using LLM architectures to learn underlying workflow sequences in clinician-EHR interactions.

Two encodings of the same logs were fine-tuned on Llama-3-8B: symbolic tokens based on fields, and natural-language descriptions of each action. The Markov comparator used only the immediately preceding action, so it served as a minimal-context benchmark.

Evaluation covered an early-period test set and two temporally distinct out-of-sample periods. The field-based LLM scored below the Markov baseline in the early test, while the word-based model scored well above it in every period.

The authors conclude that the results support developing natural-language LLM models for audit log research on task identification, workflow characterization, and safety-focused analyses.