aceUse of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study
Short-answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading and concerns about interrater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment. This study evaluated the performance and consistency of LLMs-ChatGPT, Claude Sonnet 4, and Mistral Pro-compared with those of human graders for SAQ grading in hepatology and gastroenterology. We retrospe…
Claude Sonnet 4 and GPT-4o3 graded hepatology and gastroenterology short answers close to the human majority score, which could cut the manual grading burden for medical faculty.
The authors say broader multicenter validation across domains, stakes and question formats is needed, with safeguards for bias, privacy and model drift, and that agreement does not ensure educational validity.
Evidence
- Peer-reviewedJMIR Medical Education2026-11-02
How should this claim be treated?
Truvace Impact Record TRV-2026-2355, v1: “Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study.” Truvace, 2026-11-04. /record/TRV-2026-2355 (accessed at citation time). sha256 8c45cf06160b9937…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-2355 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.