Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 11 trials finds AI teaching builds medical students' practical skills but not exam knowledge

Study title: Effectiveness of generative artificial intelligence-based teaching versus traditional teaching methods in medical education: a meta-analysis of randomized controlled trials

Evidence type
Systematic review or meta-analysis
Sample
n = 786
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2832cite or check it

Study title: Effectiveness of generative artificial intelligence-based teaching versus traditional teaching methods in medical education: a meta-analysis of randomized controlled trials

A systematic review and meta-analysis pooled 11 randomized trials with 786 medical students (395 taught with generative AI, 391 with traditional methods). Knowledge scores did not differ significantly between the two approaches (standardized mean difference 0.27, 95% CI -0.31 to 0.85).

Read the study abstract

BACKGROUND: Artificial intelligence (AI) has demonstrated remarkable capabilities across diverse medical applications, potentially revolutionizing healthcare delivery systems. This systematic review and meta-analysis investigated the comparative effectiveness of generative artificial intelligence (GAI)-based teaching methodologies versus conventional pedagogical approaches on educational outcomes among medical students. METHODS: We conducted a comprehensive literature search across multiple electronic databases,…

61Gain side

How the two sides scoreClose call

60Problem side

The gain

A meta-analysis of two randomized trials (103 medical students) found generative AI teaching produced higher practical skill scores than traditional teaching (SMD 0.63, 95% CI 0.10 to 1.16).

The problem

A meta-analysis of 11 randomized trials with 786 medical students found no significant knowledge-score difference between generative AI teaching and traditional teaching (SMD 0.27, 95% CI -0.31 to 0.85), with low-quality evidence.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Practical skills favored AI-based teaching (SMD 0.63), but that result rests on only two studies with 103 students. Students in the AI groups also reported higher satisfaction in three studies (SMD 1.18).

The authors rated the overall evidence as low quality, citing heterogeneity, thin reporting of randomization and little blinding. Long-term effects have not been measured, so the findings apply to short interventions.

Main points

  1. The review searched PubMed, Cochrane Library, EMBASE and Web of Science for studies from January 2014 to January 2025 and included 11 trials from 1,042 screened records.
  2. All included trials were published in 2024 or 2025; 8 (72.7%) were from Asian institutions, and ChatGPT was the main tool in 9 (81.8%).
  3. Knowledge scores showed no significant difference overall (p = 0.36), with very high heterogeneity (I² = 93%).
  4. Subgroup analyses showed better knowledge scores with AI in interventions longer than one week (p = 0.02) and in practice courses (p < 0.01), but little effect in theory courses.
  5. Skill scores favored AI teaching (SMD 0.63, p = 0.02) in two studies with 52 AI-taught and 51 traditionally taught students.
  6. Satisfaction was higher in the AI group across three studies (SMD 1.18, 95% CI 0.50 to 1.87).
  7. The GRADE rating was low; only one study had low risk for allocation concealment and only three reported blinding methods.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Medical students' practical skill scores with generative AI teaching compared with traditional teaching

    SMD 0.63 (95% CI 0.10 to 1.16)

    2 studies · search to January 2025

    The review’s own words
    Additionally, the meta-analysis, which involved only two studies (52 GAI-based students and 51 traditional teaching students), showed that the GAI-based teaching had significantly higher skill scores compared to the traditional teaching method (SMD 0.63, 95% CI 0.10 to 1.16; p = 0.02; I² = 43%; Fig. 3 ), under the random-effects model.

  • Medical students' teaching satisfaction with generative AI teaching compared with traditional teaching

    SMD 1.18 (95% CI 0.50 to 1.87)

    search to January 2025

    The review’s own words
    There were three studies conducted on the survey on teaching satisfaction in both student groups, and the GAI group reported higher satisfaction than the traditional group (SMD 1.18, 95% CI 0.50 to 1.87; p < 0.01; I² = 79%; Fig. 3 ), with the difference being statistically significant.

  • Medical students' knowledge scores with generative AI teaching compared with traditional teaching

    SMD 0.27 (95% CI −0.31 to 0.85)

    11 studies · 786 people · search to January 2025

    The review’s own words
    Meta-analysis with a random-effects model showed that there was no significant difference in knowledge scores between the groups (SMD 0.27, 95% CI −0.31 to 0.85; p = 0.36; I² = 93%; Fig. 3 ).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors followed PRISMA reporting and registered the protocol with INPLASY in January 2025. Two reviewers independently screened records; 1,200 were found, 158 duplicates removed, and 32 full texts assessed before 11 trials were included. Risk of bias used the Cochrane tool and evidence quality used GRADE.

Participants were mainly undergraduates, with one study including postgraduates. Five studies ran interventions longer than one week; six covered theory courses and five covered practical courses. Practice courses were defined as those held during clinical rotations or internships.

The subgroup findings suggest AI helped more in longer and practice-oriented courses, while geography and assessment timing did not change the comparison. The authors say the short, infrequent AI use in the trials may have limited results and call for standardized protocols and long-term studies.

The authors conclude AI teaching should be integrated into medical curricula, as a complement to traditional teaching, with training for educators and students.

What this doesn’t fix

The authors say statistical power may be limited by the small number of studies and modest sample sizes, substantial heterogeneity remained, and many trials poorly documented randomization and blinding.

Sources

1 on file
  1. BMC Medical Educationdoi:10.1186/s12909-025-07750-2
    Peer-reviewed2025-08-19

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Review of 11 trials finds AI teaching builds medical students' practical skills but not exam knowledgehttps://truvace.com/item/effectiveness-of-generative-artificial-intelligence-based-teaching-versus-tradit

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain A meta-analysis of two randomized trials (103 medical students) found generative AI teaching produced higher practical skill scores than traditional teaching (SMD 0.63, 95% CI 0.10 to 1.16).

    The problem A meta-analysis of 11 randomized trials with 786 medical students found no significant knowledge-score difference between generative AI teaching and traditional teaching (SMD 0.27, 95% CI -0.31 to 0.85), with low-quality evidence.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 786, from BMC Medical Education.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The authors say statistical power may be limited by the small number of studies and modest sample sizes, substantial heterogeneity remained, and many trials poorly documented randomization and blinding.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health