Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2203
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-23T06:45:37.331873Z
status: published
lens: trace
sector: health
headline: Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study
dek: Background: Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and sup…
gain_title: Large general-purpose LLMs gave stable, rubric-guided ratings of virtual-patient summary statements that agreed closely with expert raters, especially with few-shot plus chain-of-thought prompting.
problem_title: Concordance with human experts was lower on the nuanced ternary rubric components, and the smaller model showed weak repeat agreement and more invalid outputs.
trace_subject: LLM rubric-based scoring of clinical summary statements in virtual patient training
gain_reading: Large general-purpose LLMs gave stable, rubric-guided ratings of virtual-patient summary statements that agreed closely with expert raters, especially with few-shot plus chain-of-thought prompting.
gain_evidence: Few-shot+CoT prompting achieved the best concordance with human ratings for all the models
problem_reading: Concordance with human experts was lower on the nuanced ternary rubric components, and the smaller model showed weak repeat agreement and more invalid outputs.
problem_evidence: The performance on more nuanced, ternary reasoning dimensions was more variable. | Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.
quick_read: Researchers tested four language models on 122 English and German summary statements written for six internal medicine virtual patient cases, which two experts had already scored. Each model was run five times under four prompting strategies to see whether its ratings stayed stable and matched the experts.

This matters because virtual patient systems give learners little individualized feedback on their summaries, and automated scoring could make such feedback scalable. The reliability and agreement findings support that use, but accuracy on the nuanced components varied, so human judgment is still needed.
limitation: Performance was more variable on the nuanced ternary reasoning components, and the smaller local model produced more invalid outputs, so the models should support rather than replace human judgment.
tag: Dual reading
key_points: The study used 122 rubric-scored statements from 6 internal medicine virtual patient cases, rated earlier by 2 experts. | Claude was essentially deterministic across repeated runs, with ICC and Fleiss kappa of at least 0.99. | Mistral Large with few-shot plus chain-of-thought prompting reached 77% accuracy and Gwet AC1 and AC2 of 0.85. | Agreement was higher for the binary components (person and factual accuracy) than for the ternary components. | Mistral 7B had high ICCs but low Fleiss kappa (0.32-0.52) and the lowest concordance at 60% accuracy.
rundown: Statements were scored on four ternary components (semantic qualifiers, transformation, narrowing and global rating) and two binary components (factual accuracy and person). The models tested were Claude, GPT-4o, Mistral Large and Mistral 7B, under zero-shot, zero-shot with chain-of-thought, few-shot and few-shot with chain-of-thought prompts.

Reliability was measured with intraclass correlation coefficients and Fleiss kappa. Concordance with the experts was measured with accuracy and Gwet AC1 and AC2. GPT-4o and Mistral Large reached ICC of at least 0.97 and Fleiss kappa of at least 0.80 except under zero-shot.

After Mistral Large, concordance under the best prompting was 71% for Claude and 67% for GPT-4o. The authors conclude that LLMs can support scalable formative feedback on clinical reasoning within virtual patient systems but should complement rather than replace human judgment.
sources:
- peer_reviewed | JMIR Medical Education | https://doi.org/10.2196/94666 | 2026-10-21
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

14d03ef1da7b791647bbc9cf15b6040460fac7fa482504e11a80ff0fcbfcf235

previous

0000000000000000000000000000000000000000000000000000000000000000