64/100

Moderate evidence

Cite
More
EmailLinkedInX

AI scored 36 students' surgical recall close to an expert's marks, study finds

Study title: Human versus AI scoring of surgical mental practice in the FingerFix Smart® procedure

Evidence type
Controlled or validated comparison
Sample
n = 36
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Human versus AI scoring of surgical mental practice in the FingerFix Smart® procedure

Researchers tested whether generative AI could grade how well 36 medical students, all of whom had completed a hand surgery rotation, recalled placement of the FingerFix Smart dynamic external fixator. After watching an expert surgeon's video, the students described the procedure aloud, and the narrations were transcribed and scored against a 43-item checklist.

Read the study abstract

Introduction: Assessment of surgical procedural knowledge remains largely dependent on human expertise, which may limit reproducibility and scalability. This study evaluated whether generative artificial intelligence could score a surgical mental practice task from audio transcripts after video-based learning of FingerFix Smart® dynamic external fixator placement. Methods: Thirty-six medical students who had completed a rotation in a hand surgery department watched a reference video performed by an expert surgeo…

The gain

In a study of 36 medical students describing a hand-surgery fixator procedure, the fourth AI prompt scored their narrations within a mean bias of +0.92 points of an expert human grader, with an intraclass correlation of 0.940.

In brief

The human grader gave a mean score of 28.42 out of 43. After the first two AI prompts under- and then overestimated those scores, the fourth prompt averaged 29.33 out of 43 and agreed closely with the human marks.

Agreement was not perfect for individual students. The mean absolute error was 2.19 points, and the range in which most differences fell ran from -4.94 to +6.78 points. The prompts were refined and tested on the same transcripts, so the result does not show the method would hold on new students.

Main points

  1. The study involved 36 medical students who had finished a rotation in a hand surgery department.
  2. Students watched an expert surgeon's reference video, then orally reproduced the FingerFix Smart placement procedure.
  3. Narrations were transcribed and scored on a standardized 43-item checklist by an expert human and by AI.
  4. The human mean score was 28.42 (SD 9.60) out of 43; the fourth AI prompt gave 29.33 (SD 8.20).
  5. The first prompt underestimated human scores and the second overestimated them; two more prompts followed an item-by-item discrepancy analysis.
  6. Mean absolute error was 2.19 points, and Bland-Altman limits of agreement ran from -4.94 to +6.78 points.

The full story

The task tested surgical mental practice: after seeing an expert surgeon place a FingerFix Smart dynamic external fixator, each student described the steps aloud. Their transcribed narrations were scored on a 43-item checklist.

An expert human evaluator's scores served as the benchmark. AI scoring used sequentially calibrated prompts. The first prompt scored too low and the second too high, so the team analysed disagreements item by item and wrote two further prompts.

The fourth prompt came closest, at 29.33 versus 28.42 points on average, with an intraclass correlation of 0.940 (95% CI 0.884-0.969). The mean bias of +0.92 points had a confidence interval spanning zero (-0.09 to +1.93).

Individual-level differences remained, with a mean absolute error of 2.19 points and limits of agreement from -4.94 to +6.78 points. The authors rate the level of evidence as III and say independent validation is required.

Sources

1 on file
  1. Hand Surgery and Rehabilitationdoi:10.1016/j.hansur.2026.103056
    Peer-reviewed2026-11-17

Talk it over

Three questions to start on, two more if you have time
The fourth AI prompt averaged 29.33 out of 43 against the expert's 28.42, yet individual differences ran from -4.94 to +6.78 points. How would you weigh close group averages against those gaps for a single student?

Start with a question

AI scored 36 students' surgical recall close to an expert's marks, study findshttps://truvace.com/item/human-versus-ai-scoring-of-surgical-mental-practice-in-the-fingerfix-smart-proce

  1. The fourth AI prompt averaged 29.33 out of 43 against the expert's 28.42, yet individual differences ran from -4.94 to +6.78 points. How would you weigh close group averages against those gaps for a single student?

    Answer in the thread
  2. The prompts were refined and tested on the same 36 transcripts. What would you want to see from new students and a second grader before trusting this scoring?

    Answer in the thread
  3. If you were one of the 36 medical students describing the FingerFix Smart procedure aloud, how would you want an AI score to be used, and who should check it?

    Answer in the thread
If you have more time
  • Look up how the 43-item checklist could be applied by two human graders. How much would they differ from each other compared with the AI?

    Answer in the thread
  • Think of a training programme you know. Which tasks could use AI scoring of spoken answers, and which would still need an expert in the room?

    Answer in the thread

What readers say