Moderate evidence
More
AI scored surgical residents' feedback like experts did in a 1,509-comment study
Study title: Using Artificial Intelligence to Evaluate the Quality of Narrative Feedback in Surgical Entrustable Professional Activities
- Evidence type
- Observational study
- Sample
- n = 1,509
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-3033cite or check it
Study title: Using Artificial Intelligence to Evaluate the Quality of Narrative Feedback in Surgical Entrustable Professional Activities
Researchers tested whether fine-tuned language models could score the quality of written feedback given to surgical residents. They used 1,509 narrative comments from entrustable professional activity micro-assessments in one general surgery residency program, collected over the 2023-2025 academic years.
Read the study abstract
Background: Narrative feedback quality in surgical Entrustable Professional Activity (EPA) assessments is variable, limiting consistent evaluation and opportunities for improvement. We evaluated whether fine-tuned large language models (LLMs) could reproduce expert-assigned feedback quality scores and whether a simplified scoring rubric improved assessment consistency. Study design: This observational study evaluated 1,509 narrative comments from EPA micro-assessments in a general surgery residency program durin…

The gain
In an observational study of 1,509 surgical residency feedback comments, fine-tuned encoder language models matched expert quality scores with accuracy of 0.93 for evidence, 0.96 for suggestion and 0.95 for connection.
In brief
Two surgical education experts scored each comment on evidence, suggestion and connection. Encoder models reached accuracy of 0.93 for evidence, 0.96 for suggestion and 0.95 for connection, while a generative model scored 0.77 for evidence.
The experts themselves disagreed on 27.8% of evidence scores under the original rubric. A simplified rubric with one point per domain cut that disagreement to 8.5% and gave the models their best performance. The authors say the results support further evaluation of AI-assisted feedback scoring in residency programs.
Main points
- The study analysed 1,509 narrative comments from entrustable professional activity micro-assessments in a general surgery residency program during 2023-2025.
- Two surgical education experts scored comments using the Quality of Assessment for Learning rubric, which covers evidence, suggestion and connection on a 0-5 scale.
- Interrater reliability between the experts was high (κ = 0.85), yet they disagreed on 27.8% of evidence scores.
- A modified rubric awarding 1 point per domain on a 3-point scale cut evidence disagreement to 8.5%.
- Encoder and generative models scored similarly for suggestion (0.96 vs 0.94) and connection (0.95 vs 0.91).
- For evidence, the encoder model reached 0.93 accuracy against 0.77 for the generative model.
- Model performance was strongest with the modified rubric.
The full story
Feedback on surgical trainees' performance is often variable in quality, which limits consistent evaluation and chances to improve. The team asked whether fine-tuned language models could reproduce expert judgments of that quality.
They compared fine-tuned encoder models and generative models with scores from two surgical education experts, using accuracy and F1 score. Encoder models led on evidence scoring, and both model types did well on suggestion and connection.
The rubric change mattered for people as well as machines. Making evidence scoring binary in the modified rubric reduced human disagreement from 27.8% to 8.5% and improved model performance.
Sources
1 on file- Journal of the American College of Surgeonsdoi:10.1097/xcs.0000000000002297
Talk it over
The two experts disagreed on 27.8% of evidence scores under the original rubric, and a simplified rubric cut that to 8.5%. What does that tell you about using expert scores as the benchmark for AI?
AI scored surgical residents' feedback like experts did in a 1,509-comment studyhttps://truvace.com/item/using-artificial-intelligence-to-evaluate-the-quality-of-narrative-feedback-in-s
The two experts disagreed on 27.8% of evidence scores under the original rubric, and a simplified rubric cut that to 8.5%. What does that tell you about using expert scores as the benchmark for AI?
Answer in the threadEncoder models reached 0.93 accuracy on evidence while a generative model reached 0.77. Which kind of model would you choose to score feedback to surgical residents, and what would drive that choice?
Answer in the threadIf you were a surgical resident receiving written feedback, how would you want an AI quality score on your supervisors' comments to be used, and by whom?
Answer in the thread
If you have more time
All 1,509 comments came from one general surgery residency program over 2023-2025. What would you want to check before using the models in another program?
Answer in the threadThe simplified rubric gave one point per domain on a 3-point scale. What might be gained or lost by scoring feedback quality in such a coarse way?
Answer in the thread
ace
What readers say