Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1785
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-18T06:53:51.908565Z
status: published
lens: trace
sector: education
headline: How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability
dek: Background In the health professions, artificial intelligence simulation-based experiences (AI-SBEs) are increasingly used to evaluate learner performance and provide feedback. Benefits of AI-SBEs include providing instantaneous feedback, allowing more practice opportunities than feasible using traditional methods, and reducing educator burden. However, methods for calibrating AI feedback are underdeveloped. Methods This study evaluated the feedback provided by a generative AI-based software program (SimConverse…
gain_title: After prompt engineering, AI scoring of a content-specific checklist item matched expert scoring far better, and agreement was good or very good on most checklist items.
problem_title: Three checklist items that needed broader reading of the transcript kept showing weaker AI-expert agreement even after tuning.
trace_subject: AI scoring of medical students' weight-bias behaviors in simulated patient interactions
gain_reading: After prompt engineering, AI scoring of a content-specific checklist item matched expert scoring far better, and agreement was good or very good on most checklist items.
gain_evidence: agreement improved after prompt engineering (final Gwet's AC1=0.89) | expert and AI agreement was consistently good or very good for 7 items across all rounds
problem_reading: Three checklist items that needed broader reading of the transcript kept showing weaker AI-expert agreement even after tuning.
problem_evidence: Three other items had persistent poor agreement (final Gwet's AC1=0.42, 0.66, 0.64) | These items required broader contextual assessment of the transcripts.
quick_read: Researchers tested how closely a generative AI simulation program, SimConverse, scored simulated patient conversations compared with a human expert using an 11-item checklist on weight-bias-related clinical behaviors. They revised the AI's rubric instructions in rounds, then piloted the tool with 60 medical students and checked a sample of their transcripts.

The results suggest AI feedback in health-professions training cannot be used as-is and needs educator and researcher oversight to be calibrated. Tuning worked for items about specific statements but not for items requiring judgment of overall context, so it remains unclear how well such tools handle more complex rubric items.
limitation: Three context-dependent items kept weaker agreement after tuning, so the AI scoring was not reliably calibrated across the whole checklist.
tag: Dual reading
key_points: Agreement was good or very good (Gwet's AC1 of 0.71 or higher) for 7 of 11 checklist items across all rounds. | One item rose from poor agreement (AC1 0.56) to 0.89 after prompt engineering; it asked whether students asked if weight loss was a priority. | Three items still showed weaker agreement (AC1 0.42, 0.66, 0.64) and needed broader contextual assessment. | Most open-ended student comments about the AI simulation experience were positive.
rundown: In phase one, four authors and four volunteers generated 31 interactions in SimConverse. An author expert in the Weight Sensitivity Instrument scored the downloaded transcripts by hand, and percent agreement and Gwet's AC1 were calculated against the AI's scoring to find rubric instructions needing revision.

After the rubric was revised, four authors produced 10 more interactions for comparison. In phase two, 60 medical students used the tool as part of a larger intervention study, and 30 of their transcripts were randomly sampled for AI-human agreement checks.

The authors conclude that tuning was more successful for content-specific items than for context-dependent items, and they call for future work on best practices for prompt refinement for complex rubric items.
sources:
- peer_reviewed | Advances in Simulation | https://doi.org/10.1186/s41077-026-00464-5 | 2026-10-07
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

63e0f21e64838cc12d822157742bf49ff3471297e2bfa528835c5ef53d7330c0

previous

0000000000000000000000000000000000000000000000000000000000000000