aceHow does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability
Background In the health professions, artificial intelligence simulation-based experiences (AI-SBEs) are increasingly used to evaluate learner performance and provide feedback. Benefits of AI-SBEs include providing instantaneous feedback, allowing more practice opportunities than feasible using traditional methods, and reducing educator burden. However, methods for calibrating AI feedback are underdeveloped. Methods This study evaluated the feedback provided by a generative AI-based software program (SimConverse…
After prompt engineering, AI scoring of a content-specific checklist item matched expert scoring far better, and agreement was good or very good on most checklist items.
Three checklist items that needed broader reading of the transcript kept showing weaker AI-expert agreement even after tuning.
Three context-dependent items kept weaker agreement after tuning, so the AI scoring was not reliably calibrated across the whole checklist.
Evidence
- Peer-reviewedAdvances in Simulation2026-10-07
How should this claim be treated?
Truvace Impact Record TRV-2026-1785, v1: “How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability.” Truvace, 2026-10-18. /record/TRV-2026-1785 (accessed at citation time). sha256 63e0f21e64838cc1…
Calibration history
Every change to this record since certification, in the open. None yet — the reading has held since it entered the record.
Certified into the record
How to verify without trusting this page
Fetch the canonical text of any version from /api/record/TRV-2026-1785 and hash it yourself — for example shasum -a 256 on the saved canonical field. The result must equal content_hash, and each version’s text ends with prev:followed by the prior version’s hash (version 1 chains to 64 zeros). If a single character of any version had been altered since certification, the chain would not reproduce.