Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 66 trials finds AI tutors lift health students' scores, but evidence is weak

Study title: The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials

Evidence type
Systematic review or meta-analysis
Sample
n = 4,911
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2810cite or check it

Study title: The Effectiveness of Artificial Intelligence in Undergraduate Health Professions Education: Systematic Review and Meta-Analysis of Randomized Controlled Trials

A systematic review and meta-analysis pooled 66 randomized trials with 4,911 undergraduate health professions students, published 2020 to 2026, comparing AI teaching tools with standard education. The largest evidence base was AI chatbot personalized learning aids, which improved satisfaction (SMD 0.93), confidence (SMD 0.91) and theoretical knowledge (SMD 0.53).

Read the study abstract

Background: Health professions education faces increasing challenges from rising health care complexity, pedagogical shifts, and constrained curricular space, and rapidly expanding knowledge and technological advances. While artificial intelligence (AI) shows promise for transforming health professions education, evidence of its effectiveness remains unclear. Objective: This study synthesized evidence from randomized controlled trials (RCTs) on the effectiveness of AI in undergraduate health professions educatio…

61Gain side

How the two sides scoreProblem scores higher

67Problem side

The gain

A meta-analysis of 66 randomized trials (4,911 undergraduate health students) found AI chatbot-based personalized learning aids beat standard teaching on theoretical knowledge (SMD 0.53, 12 studies, 955 participants), though certainty was very low.

The problem

A meta-analysis of 66 randomized trials of AI in undergraduate health education found most had high risk of bias, certainty ranged from low to very low, and no trial measured real-life practice or patient outcomes.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Every one of those findings carried very low certainty. Other AI types, including content generators, rule-based chatbots and non-chatbot adaptive learning platforms, had mostly favorable estimates but wide intervals that often included no effect, and prediction intervals frequently crossed the null.

The authors conclude the evidence is insufficient to guide educational practice and that effects are inconsistent and not reliably reproducible. They recommend AI applications be used cautiously and on a trial basis.

Main points

  1. The review searched five databases up to January 26, 2026, screening 39,783 records and including 66 trials (74 records).
  2. Satisfaction improved with chatbot-based learning aids (SMD 0.93, 7 studies, 430 participants) and confidence (SMD 0.91, 7 studies, 609 participants).
  3. Theoretical knowledge rose by SMD 0.53 across 12 studies and 955 participants, with heterogeneity of 86%.
  4. Subcategory analyses yielded 48 comparisons across 7 outcome domains, and most studies had high risk of bias from poor allocation concealment and blinding.
  5. Medicine students made up 53% of trials, nursing 22.7% and dentistry 13.6%; China (14 trials) and Turkey (11) contributed most.
  6. LLM tools were used in 38 trials (57.6%), mainly ChatGPT, and 28 trials used non-LLM systems such as convolutional neural networks and rule-based chatbots.
  7. All trials measured only reaction or knowledge; none assessed real-life practice or patient health outcomes.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Theoretical knowledge of students using LLM-based personalized learning aids in health professions education

    SMD 0.53 (95% CI 0.13 to 0.94)

    12 studies · 955 people · search to January 26, 2026

    The review’s own words
    theoretical knowledge (SMD 0.53, 95% CI 0.13-0.94; 12 studies; 955 participants; I²=86%)

  • Student satisfaction with LLM-based personalized learning aids in undergraduate health professions education

    SMD 0.93 (95% CI 0.40 to 1.46)

    7 studies · 430 people · search to January 26, 2026

    The review’s own words
    Large language model (LLM)-based personalized learning aids comprised the largest evidence base and showed positive effects for satisfaction (standardized mean difference [SMD] 0.93, 95% CI 0.40-1.46; 7 studies; 430 participants; I²=74%)

  • Student confidence with LLM-based personalized learning aids in undergraduate health professions education

    SMD 0.91 (95% CI 0.54 to 1.29)

    7 studies · 609 people · search to January 26, 2026

    The review’s own words
    confidence (SMD 0.91, 95% CI 0.54-1.29; 7 studies; 609 participants; I²=64%)

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The review included randomized, crossover and cluster-randomized trials of undergraduate students in medicine, pharmacy, nursing and allied health, excluding postgraduate training and quasi-experimental studies. Of 177 papers assessed in full text, 73 were excluded for design, population or non-AI interventions, and 30 ongoing studies were identified.

Of the 66 trials, 58 were parallel RCTs, 7 were crossover trials and 1 was cluster-randomized. Median sample size was 63 participants, ranging from 16 to 426. AI served mainly as a teaching tool in 54 trials, and 36.4% of studies lasted a single session of under 3 hours.

Controls usually received standard curriculum methods such as lectures, web tools and human-led simulation (39 trials), or the same content without AI (18 trials). Outcomes were mostly measured immediately; only five studies followed students from 1 week to 3 months.

Risk of bias was rated with the ROBUST-RCT tool and certainty with GRADE. The authors used an LLM, Claude Sonnet, for preliminary data extraction, which a human verified against the full texts.

What this doesn’t fix

The authors say high risk of bias, heterogeneity, imprecision and the absence of higher-level outcomes limit conclusions, and effects were inconsistent and not reliably reproducible.

Sources

1 on file
  1. JMIR Medical Educationdoi:10.2196/88933
    Peer-reviewed2026-03-30

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Review of 66 trials finds AI tutors lift health students' scores, but evidence is weakhttps://truvace.com/item/the-effectiveness-of-artificial-intelligence-in-undergraduate-health-professions

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain A meta-analysis of 66 randomized trials (4,911 undergraduate health students) found AI chatbot-based personalized learning aids beat standard teaching on theoretical knowledge (SMD 0.53, 12 studies, 955 participants), though certainty was very low.

    The problem A meta-analysis of 66 randomized trials of AI in undergraduate health education found most had high risk of bias, certainty ranged from low to very low, and no trial measured real-life practice or patient outcomes.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 4,911, from JMIR Medical Education.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The authors say high risk of bias, heterogeneity, imprecision and the absence of higher-level outcomes limit conclusions, and effects were inconsistent and not reliably reproducible.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education