Moderate evidence
More
Pilot study suggests AI chart summaries kept doctors' accuracy while speeding frequent users
Study title: Verifiable summarization of electronic health records using large language models to support chart review
- Evidence type
- Controlled or validated comparison
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2919cite or check it
Study title: Verifiable summarization of electronic health records using large language models to support chart review
Researchers built an extract-then-abstract approach using large language models to turn longitudinal clinical notes into problem-based admission summaries for patients admitted with heart failure. The summaries were guided by disease-specific "summary bundles" that prioritize clinically relevant information.
Read the study abstract
Objectives: Information overload in electronic health records hampers clinicians' ability to efficiently extract and synthesize critical information from a patient's longitudinal health record, leading to increased cognitive burden and delays in care. This study explores the potential of large language models (LLMs) to address this challenge by generating problem-based admission summaries for patients admitted with heart failure, a leading cause of hospitalization worldwide. Materials and methods: We developed a…
The gain
In a pilot study of physicians reviewing heart failure admissions, accuracy answering clinical questions was 88.0% with LLM-generated summaries versus 86.4% with standard chart review, and frequent summary use was linked to faster completion (P = .002).
In brief
In a mixed-methods evaluation on real-world clinical notes, physicians answered patient-specific clinical questions using either the summaries or standard chart review. Accuracy was 88.0% with summaries and 86.4% without, so accuracy held steady. Access to summaries did not significantly cut overall completion time, but physicians who used them often finished faster (P = .002).
The work is an exploratory pilot, and physicians varied in how well they used the summaries and in the formats they preferred. That points to a need for customizable summaries that fit individual clinician workflows.
Main points
- The study targeted heart failure, a leading cause of hospitalization worldwide, using real-world longitudinal clinical notes.
- The method extracts relevant information first and then abstracts it, guided by disease-specific summary bundles.
- Physicians answered patient-specific clinical questions with the summaries and with standard chart review for comparison.
- Accuracy was 88.0% with summaries versus 86.4% without.
- Summary access alone did not significantly reduce overall questionnaire completion time.
- All physicians said they were likely or very likely to use the summaries in practice, and 87.5% said the summaries would save them time.
- Physicians differed in how effectively they used the summaries and in their preferred format.
The full story
The study addresses information overload in electronic health records, which the authors say raises clinicians' cognitive burden and can delay care. It focuses on admission summaries for heart failure patients.
The summaries come from an extract-then-abstract approach built around disease-specific summary bundles. Evaluation used real-world clinical notes and compared physicians' answers to patient-specific clinical questions using LLM summaries versus standard chart review.
Results showed accuracy of 88.0% with summaries versus 86.4% without. Overall completion time did not drop significantly with summary access, though frequent use was associated with faster completion (P = .002). Individual physicians varied in how well they used the summaries.
In self-reports, all physicians said they were likely or very likely to use the summaries clinically and 87.5% said they would save time. Format preferences varied, which the authors say supports customizable summaries and user-centered evaluation.
Sources
1 on file- Journal of the American Medical Informatics Associationdoi:10.1093/jamia/ocag181
Talk it over
Accuracy was 88.0% with LLM summaries and 86.4% with standard chart review, and overall completion time did not drop significantly. How do you weigh those results against the faster work of frequent users (P = .002)?
Pilot study suggests AI chart summaries kept doctors' accuracy while speeding frequent usershttps://truvace.com/item/verifiable-summarization-of-electronic-health-records-using-large-language-model
Accuracy was 88.0% with LLM summaries and 86.4% with standard chart review, and overall completion time did not drop significantly. How do you weigh those results against the faster work of frequent users (P = .002)?
Answer in the threadAll physicians said they were likely or very likely to use the summaries, and 87.5% said the summaries would save them time. How far can those self-reports tell us about how the summaries work in practice?
Answer in the threadPhysicians varied in how well they used the summaries and in the formats they preferred. Who should decide how customizable summaries are built for heart failure admissions, and what should they test?
Answer in the thread
If you have more time
Frequent users finished faster, but were they already faster reviewers? What comparison would separate the effect of the summaries from the habits of the physicians?
Answer in the threadThe summaries came from real-world longitudinal clinical notes on heart failure admissions. What would change in a hospital you know if a physician had to check a summary against the chart?
Answer in the thread
ace
What readers say