Truace
Subscribe

AI scoring of medical students' weight-bias behaviors in simulated patient interactions

Source article: How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability

Background In the health professions, artificial intelligence simulation-based experiences (AI-SBEs) are increasingly used to evaluate learner performance and provide feedback. Benefits of AI-SBEs include providing instantaneous feedback, allowing more practice opportunities than feasible using traditional methods, and reducing educator burden. However, methods for calibrating AI feedback are underdeveloped. Methods This study evaluated the feedback provided by a generative AI-based software program (SimConverse…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1785· cite & verify
How does AI-generated assessment compare to expert assessment in simulation-based experiences: Analyzing AI-human interrater reliability
Hospital Universitari Doctor Peset, València 11 by 19Tarrestnom65. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0
Trace impact readingContested
P 65The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers tested how closely a generative AI simulation program, SimConverse, scored simulated patient conversations compared with a human expert using an 11-item checklist on weight-bias-related clinical behaviors. They revised the AI's rubric instructions in rounds, then piloted the tool with 60 medical students and checked a sample of their transcripts.

The results suggest AI feedback in health-professions training cannot be used as-is and needs educator and researcher oversight to be calibrated. Tuning worked for items about specific statements but not for items requiring judgment of overall context, so it remains unclear how well such tools handle more complex rubric items.

Main points

  1. Agreement was good or very good (Gwet's AC1 of 0.71 or higher) for 7 of 11 checklist items across all rounds.
  2. One item rose from poor agreement (AC1 0.56) to 0.89 after prompt engineering; it asked whether students asked if weight loss was a priority.
  3. Three items still showed weaker agreement (AC1 0.42, 0.66, 0.64) and needed broader contextual assessment.
  4. Most open-ended student comments about the AI simulation experience were positive.

The gain

After prompt engineering, AI scoring of a content-specific checklist item matched expert scoring far better, and agreement was good or very good on most checklist items.

The problem

Three checklist items that needed broader reading of the transcript kept showing weaker AI-expert agreement even after tuning.

The rundown

In phase one, four authors and four volunteers generated 31 interactions in SimConverse. An author expert in the Weight Sensitivity Instrument scored the downloaded transcripts by hand, and percent agreement and Gwet's AC1 were calculated against the AI's scoring to find rubric instructions needing revision.

After the rubric was revised, four authors produced 10 more interactions for comparison. In phase two, 60 medical students used the tool as part of a larger intervention study, and 30 of their transcripts were randomly sampled for AI-human agreement checks.

The authors conclude that tuning was more successful for content-specific items than for context-dependent items, and they call for future work on best practices for prompt refinement for complex rubric items.

What this doesn’t fix

Three context-dependent items kept weaker agreement after tuning, so the AI scoring was not reliably calibrated across the whole checklist.

Sources

  1. Peer-reviewedAdvances in Simulation2026-10-07

The debate