Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2355
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-04T00:01:41.004531Z
status: published
lens: g_space
sector: education
headline: Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study
dek: Short-answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading and concerns about interrater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment. This study evaluated the performance and consistency of LLMs-ChatGPT, Claude Sonnet 4, and Mistral Pro-compared with those of human graders for SAQ grading in hepatology and gastroenterology. We retrospe…
gain_title: Claude Sonnet 4 and GPT-4o3 graded hepatology and gastroenterology short answers close to the human majority score, which could cut the manual grading burden for medical faculty.
problem_title: (none)
trace_subject: (none)
gain_reading: Claude Sonnet 4 and GPT-4o3 graded hepatology and gastroenterology short answers close to the human majority score, which could cut the manual grading burden for medical faculty.
gain_evidence: Claude Sonnet 4 and GPT-4o3 demonstrated near-human-level performance when compared with the majority vote of human evaluators
problem_reading: (none)
problem_evidence: (none)
quick_read: Researchers compared six large language models with three human graders on 902 student answers to 12 short-answer questions from hepatology and gastroenterology exams at Sorbonne University in Paris, collected from 2023 to 2025. Each model and each human graded every response twice, using the majority human score as the reference.

The results suggest AI could ease the time-intensive marking of reasoning-style questions in medical schools, but only for well-performing models. The authors stress that validation across other subjects, stakes and question formats is still missing, and that AI should support rather than replace faculty graders.
limitation: The authors say broader multicenter validation across domains, stakes and question formats is needed, with safeguards for bias, privacy and model drift, and that agreement does not ensure educational validity.
tag: Evidence-backed gain
key_points: Human graders agreed almost perfectly with each other (QWK 0.90-0.92). | Claude Sonnet 4 reached QWK 0.88-0.90 against the human majority score. | GPT-4o3 reached QWK 0.82-0.89, while GPT-5 showed substantial agreement at 0.77-0.79. | Claude Sonnet 4 was the most self-consistent model (QWK 0.93), approaching the best human intrarater reliability. | Prompt design choices such as number of examples or minimum word count did not correlate with agreement.
rundown: The study retrospectively analyzed 902 student responses to 12 short-answer questions. Two hepatogastroenterologists and one radiologist each graded all answers twice, and their majority consensus formed the reference standard.

Six models were tested: four ChatGPT versions (GPT-4o, GPT-4.5, GPT-4o3, GPT-5), Claude Sonnet 4 and Mistral Pro. Each graded every response twice using standardized prompts with the question, model answer, grading guidelines and scoring rubric.

Agreement was measured with quadratic weighted kappa, Cohen kappa, intraclass correlation and mean absolute error. Mistral Pro and older ChatGPT versions fell below a QWK of 0.63, showing wide variation between models.
sources:
- peer_reviewed | JMIR Medical Education | https://doi.org/10.2196/89445 | 2026-11-02
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

8c45cf06160b9937e34b2e3687f8024de198d2b06adc766bf9f437ba712eb359

previous

0000000000000000000000000000000000000000000000000000000000000000