70/100

Moderate evidence

Cite
More
EmailLinkedInX

Medical students scored lower with an AI patient than a human one in a 48-student trial

Study title: Medical Interview Performance Using an AI-Based Standardized Patient vs a Human Standardized Patient: Randomized Crossover Trial

Evidence type
Randomized controlled trial
Sample
n = 48
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Medical Interview Performance Using an AI-Based Standardized Patient vs a Human Standardized Patient: Randomized Crossover Trial

A randomized crossover trial at Chiba University Hospital in Japan compared a text-based simulated patient built on GPT-4o with a human simulated patient. Forty-eight fifth-year medical students interviewed both, and two general medicine specialists rated them. Students scored 23.9 on selected Mini-CEX items with the human patient and 22.7 with the AI patient.

Read the study abstract

Background: In Japan, standardized patients (SPs) have traditionally been recruited to teach medical interview competencies. Recent advances in large language models have enabled increasingly natural conversational interactions, raising the possibility of using AI-based SPs (AI-SPs) for medical interview training. However, medical interview performance during interactions with AI-SPs compared with human SPs remains insufficiently understood. Objective: This randomized single-center crossover trial aimed to compa…

60Gain side

How the two sides scoreClose call

59Problem side

The gain

In a randomized crossover trial of 48 Chiba University medical students, self-rated active engagement in history taking was higher with a GPT-4o simulated patient (mean 6.2) than with a human simulated patient (mean 5.8).

The problem

In a randomized crossover trial of 48 Chiba University medical students, interview scores were lower with a GPT-4o simulated patient than with a human simulated patient (mean 22.7 vs 23.9 on selected Mini-CEX items; P=.001).

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Scores for history taking and overall clinical competence were higher in the human condition. Students rated learning satisfaction and rapport about the same in both conditions, but said they were more actively engaged in history taking with the AI patient (6.2 vs 5.8 on a 7-point scale).

The authors suggest the AI patient could complement human simulated patients for interview practice. They note the study measured performance in simulated encounters, not later learning or performance with real patients.

Main points

  1. Participants were fifth-year medical students on clinical clerkships at Chiba University Hospital; 48 were analyzed.
  2. Each student completed two cases, depression and appendicitis, with both the GPT-4o text-based patient and a human simulated patient, in randomized order with a washout of at least 3 days.
  3. The mean Mini-CEX-based total score was 23.9 (SD 1.7) with the human patient versus 22.7 (SD 1.7) with the AI patient, a difference of 1.24 (P=.001).
  4. Learning satisfaction was similar: 5.9 for the human patient and 6.0 for the AI patient on a 7-point scale.
  5. Perceived rapport was similar: 5.4 for the human patient and 5.3 for the AI patient.
  6. Self-rated active engagement in history taking was higher with the AI patient: 6.2 versus 5.8 (P=.009).
  7. Interrater reliability between the two raters was 0.76 (95% CI 0.66-0.83).

The full story

The trial used a single-center randomized crossover design. Students were randomized to the AI patient first or the human patient first, and each did two cases in both conditions. AI interviews were text chats with a physician giving feedback after both cases; human interviews were video recorded with feedback from the physician portraying the patient immediately after each interview.

Two general medicine specialists independently rated performance using selected Mini-CEX items. The human patient condition scored higher on total score, history taking and overall clinical competence, with interrater reliability of 0.76.

Students' self-ratings differed from the specialist ratings. Learning satisfaction and rapport did not differ significantly, while active engagement in history taking was higher with the AI patient. The authors say the text-only format and missing nonverbal cues may have contributed to the performance gap.

What this doesn’t fix

The study assessed performance during simulated encounters rather than later learning gains or performance with real patients, and the AI patient was text-based with no nonverbal cues.

Sources

1 on file
  1. JMIR Medical Educationdoi:10.2196/111030
    Peer-reviewed2026-11-16

Talk it over

Three questions to start on, two more if you have time
Two general medicine specialists scored students lower with the GPT-4o patient (22.7 vs 23.9), yet the students rated their own engagement higher (6.2 vs 5.8). How would you weigh those two results against each other?

Start with a question

Medical students scored lower with an AI patient than a human one in a 48-student trialhttps://truvace.com/item/medical-interview-performance-using-an-ai-based-standardized-patient-vs-a-human-

  1. Two general medicine specialists scored students lower with the GPT-4o patient (22.7 vs 23.9), yet the students rated their own engagement higher (6.2 vs 5.8). How would you weigh those two results against each other?

    Answer in the thread
  2. The AI patient was text-based, with no nonverbal cues, and the study measured simulated encounters only. What can this design tell us about how students would later do with real patients?

    Answer in the thread
  3. If you were one of the fifth-year students at Chiba University Hospital, which would you choose for interview practice, the GPT-4o patient or the human simulated patient, and why?

    Answer in the thread
If you have more time
  • The authors suggest the AI patient could complement human simulated patients. What mix of the two would you want to see tested, and what outcome would settle it?

    Answer in the thread
  • Each student did two cases, depression and appendicitis, with a washout of at least 3 days. How might other cases or a voice-based AI patient change the gap?

    Answer in the thread

What readers say