Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1423
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-15T06:55:20.225416Z
status: published
lens: p_space
sector: science
headline: A language model framework for sequence modeling of audit logs to characterize clinician-electronic health record interactions
dek: Objective: Electronic health record (EHR) audit logs capture clinician-EHR interaction patterns, but most audit log research relies on aggregated measures (eg, total time). We investigated how audit logs could be modeled using large language model (LLM) architectures to learn underlying workflow sequences during clinician-EHR interactions.
Materials and methods: Using >295 million EHR-based audit log actions from inpatient settings spanning 2019 to 2024, we fine-tuned Llama-3-8B under 2 encodings: (1) symbolic (…
gain_title: (none)
problem_title: Next-action prediction accuracy of the word-based model declined in later out-of-sample periods, indicating temporal drift in the learned workflow patterns.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: Next-action prediction accuracy of the word-based model declined in later out-of-sample periods, indicating temporal drift in the learned workflow patterns.
problem_evidence: although accuracy declined over time
quick_read: Researchers used more than 295 million inpatient audit log actions recorded between 2019 and 2024 to fine-tune Llama-3-8B to predict a clinician's next action in the electronic health record. They compared two encodings, symbolic field-based tokens and word-based natural-language descriptions, against a first-order Markov baseline.

The word-based model was the most accurate and held up better than the baseline on later out-of-sample periods, though its accuracy fell over time. The findings point toward using natural-language LLMs to study clinical workflows and safety, but the work reports only next-action prediction accuracy rather than effects on care or on clinicians.
limitation: Accuracy of the word-based model declined across the later out-of-sample periods, so the learned patterns are subject to temporal drift.
tag: Evidence-backed problem
key_points: Training data comprised more than 295 million EHR audit log actions from inpatient settings, 2019 to 2024. | In the early-period test set, the word-based LLM scored 0.7473 accuracy versus 0.3464 for the field-based LLM and 0.4230 for the Markov baseline. | Out-of-sample accuracy for the word-based LLM was 0.6555 in OOS-1 and 0.6286 in OOS-2, versus 0.3139 and 0.2720 for the Markov baseline. | The authors suggest longer sequential context and word-based action descriptions may improve audit log sequence modeling.
rundown: Most audit log research has relied on aggregated measures such as total time spent in the EHR. This study instead modeled the order of actions, using LLM architectures to learn underlying workflow sequences in clinician-EHR interactions.

Two encodings of the same logs were fine-tuned on Llama-3-8B: symbolic tokens based on fields, and natural-language descriptions of each action. The Markov comparator used only the immediately preceding action, so it served as a minimal-context benchmark.

Evaluation covered an early-period test set and two temporally distinct out-of-sample periods. The field-based LLM scored below the Markov baseline in the early test, while the word-based model scored well above it in every period.

The authors conclude that the results support developing natural-language LLM models for audit log research on task identification, workflow characterization, and safety-focused analyses.
sources:
- peer_reviewed | Journal of the American Medical Informatics Association | https://doi.org/10.1093/jamia/ocag163 | 2026-10-12
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

9d774e317cfd5c190dc27c4a55c7c3d48ac8f8a84d46652a9a12a803fb44a322

previous

0000000000000000000000000000000000000000000000000000000000000000