aceTest-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study
Background: Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and sup…
Large general-purpose LLMs gave stable, rubric-guided ratings of virtual-patient summary statements that agreed closely with expert raters, especially with few-shot plus chain-of-thought prompting.
Concordance with human experts was lower on the nuanced ternary rubric components, and the smaller model showed weak repeat agreement and more invalid outputs.
Performance was more variable on the nuanced ternary reasoning components, and the smaller local model produced more invalid outputs, so the models should support rather than replace human judgment.
Evidence
- Peer-reviewedJMIR Medical Education2026-10-21
How should this claim be treated?
Truvace Impact Record TRV-2026-2203, v1: “Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study.” Truvace, 2026-10-23. /record/TRV-2026-2203 (accessed at citation time). sha256 14d03ef1da7b7916…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-2203 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.