AI chatbots grade medical student exam answers nearly as consistently as doctors

Study title: Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study

Researchers compared six large language models with three human graders on 902 student answers to 12 short-answer questions from hepatology and gastroenterology exams at Sorbonne University in Paris, collected from 2023 to 2025. Each model and each human graded every response twice, using the majority human score as the reference.

Read the study abstract

Short-answer questions (SAQs) effectively assess higher-order reasoning skills in medical education but face challenges in scalability due to time-intensive manual grading and concerns about interrater reliability. Recent advances in large language models (LLMs) may offer a solution through automated assessment. This study evaluated the performance and consistency of LLMs-ChatGPT, Claude Sonnet 4, and Mistral Pro-compared with those of human graders for SAQ grading in hepatology and gastroenterology. We retrospe…

By the Truvace deskNovember 4, 20261 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 902 · 1 independent sourcePeer-reviewedTRV-2026-2355· cite & verify

Use of Large Language Models for Automated Short-Answer Question Grading in Second-Cycle Medical Education: Performance and Consistency Evaluation Study — "Hinigaran Doctors Hospital Incorporated" by Mahiwagangdilaw is licensed under CC BY 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/.

In brief

The results suggest AI could ease the time-intensive marking of reasoning-style questions in medical schools, but only for well-performing models. The authors stress that validation across other subjects, stakes and question formats is still missing, and that AI should support rather than replace faculty graders.

Main points

Human graders agreed almost perfectly with each other (QWK 0.90-0.92).
Claude Sonnet 4 reached QWK 0.88-0.90 against the human majority score.
GPT-4o3 reached QWK 0.82-0.89, while GPT-5 showed substantial agreement at 0.77-0.79.
Claude Sonnet 4 was the most self-consistent model (QWK 0.93), approaching the best human intrarater reliability.
Prompt design choices such as number of examples or minimum word count did not correlate with agreement.

The gain

Claude Sonnet 4 and GPT-4o3 graded hepatology and gastroenterology short answers close to the human majority score, which could cut the manual grading burden for medical faculty.

The rundown

The study retrospectively analyzed 902 student responses to 12 short-answer questions. Two hepatogastroenterologists and one radiologist each graded all answers twice, and their majority consensus formed the reference standard.

Six models were tested: four ChatGPT versions (GPT-4o, GPT-4.5, GPT-4o3, GPT-5), Claude Sonnet 4 and Mistral Pro. Each graded every response twice using standardized prompts with the question, model answer, grading guidelines and scoring rubric.

Agreement was measured with quadratic weighted kappa, Cohen kappa, intraclass correlation and mean absolute error. Mistral Pro and older ChatGPT versions fell below a QWK of 0.63, showing wide variation between models.