Truace
Subscribe

LLM rubric-based scoring of clinical summary statements in virtual patient training

Source article: Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study

Background: Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and sup…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-2203· cite & verify
Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study
Hospital Universitari Doctor Peset, València 09 by 19Tarrestnom65. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0
Trace impact readingContested
P 69The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 65The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers tested four language models on 122 English and German summary statements written for six internal medicine virtual patient cases, which two experts had already scored. Each model was run five times under four prompting strategies to see whether its ratings stayed stable and matched the experts.

This matters because virtual patient systems give learners little individualized feedback on their summaries, and automated scoring could make such feedback scalable. The reliability and agreement findings support that use, but accuracy on the nuanced components varied, so human judgment is still needed.

Main points

  1. The study used 122 rubric-scored statements from 6 internal medicine virtual patient cases, rated earlier by 2 experts.
  2. Claude was essentially deterministic across repeated runs, with ICC and Fleiss kappa of at least 0.99.
  3. Mistral Large with few-shot plus chain-of-thought prompting reached 77% accuracy and Gwet AC1 and AC2 of 0.85.
  4. Agreement was higher for the binary components (person and factual accuracy) than for the ternary components.
  5. Mistral 7B had high ICCs but low Fleiss kappa (0.32-0.52) and the lowest concordance at 60% accuracy.

The gain

Large general-purpose LLMs gave stable, rubric-guided ratings of virtual-patient summary statements that agreed closely with expert raters, especially with few-shot plus chain-of-thought prompting.

The problem

Concordance with human experts was lower on the nuanced ternary rubric components, and the smaller model showed weak repeat agreement and more invalid outputs.

The rundown

Statements were scored on four ternary components (semantic qualifiers, transformation, narrowing and global rating) and two binary components (factual accuracy and person). The models tested were Claude, GPT-4o, Mistral Large and Mistral 7B, under zero-shot, zero-shot with chain-of-thought, few-shot and few-shot with chain-of-thought prompts.

Reliability was measured with intraclass correlation coefficients and Fleiss kappa. Concordance with the experts was measured with accuracy and Gwet AC1 and AC2. GPT-4o and Mistral Large reached ICC of at least 0.97 and Fleiss kappa of at least 0.80 except under zero-shot.

After Mistral Large, concordance under the best prompting was 71% for Claude and 67% for GPT-4o. The authors conclude that LLMs can support scalable formative feedback on clinical reasoning within virtual patient systems but should complement rather than replace human judgment.

What this doesn’t fix

Performance was more variable on the nuanced ternary reasoning components, and the smaller local model produced more invalid outputs, so the models should support rather than replace human judgment.

Sources

  1. Peer-reviewedJMIR Medical Education2026-10-21

The debate