aceModerate evidence
More
Review of 14 studies finds AI teaching tools may lift medical students' knowledge scores
Study title: Effectiveness of large language model-based teaching interventions in medical education: a systematic review and stratified meta-analysis of 14 studies
- Evidence type
- Systematic review or meta-analysis
- Sample
- n = 1,207
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2771cite or check it
Study title: Effectiveness of large language model-based teaching interventions in medical education: a systematic review and stratified meta-analysis of 14 studies
A systematic review and meta-analysis compared teaching built on large language models with traditional teaching for medical learners. After searching four databases through 31 March 2026, the authors screened 8,103 records and included 14 studies covering 1,207 analyzable learners.
Read the study abstract
BACKGROUND: In recent years, artificial intelligence has been adopted as an innovative educational approach and may substantially reshape health-care delivery systems. This systematic review and stratified meta-analysis compared large language model (LLM)-based teaching interventions with traditional teaching approaches in medical education. METHODS: Based on a four-database search conducted through 31 March 2026, we included comparative studies and feasibility studies evaluating LLM-based teaching tools in medi…
The gain
A systematic review of 14 studies with 1,207 medical learners found AI chatbot-based teaching was linked to higher knowledge scores (SMD 0.80, 95% CI 0.07 to 1.53, 5 studies) than traditional teaching, though certainty was low.
In brief
Pooled results favoured AI-based teaching for structured clinical performance (SMD 3.39, 95% CI 1.43 to 5.35, from 2 studies with 38 vs. 39 participants) and for strict knowledge acquisition (SMD 0.80, 95% CI 0.07 to 1.53, from 5 studies). Delayed retention showed no clear difference (SMD 0.15, 95% CI -0.50 to 0.80, 2 studies).
Certainty of evidence ranged from very low to low across pooled domains. The authors describe the findings as cautious signals rather than definitive estimates, with the clinical performance figure possibly inflated by small-study effects.
Main points
- Four databases (PubMed, Web of Science, Embase, Cochrane Library) yielded 8,103 records; 25 full texts were assessed and 14 studies were included.
- Structured clinical performance: SMD 3.39 (95% CI 1.43 to 5.35) from 2 studies with 38 vs. 39 participants.
- Strict knowledge acquisition: SMD 0.80 (95% CI 0.07 to 1.53) across 5 studies.
- Delayed retention showed no clear difference: SMD 0.15 (95% CI -0.50 to 0.80) in 2 studies.
- An exploratory pooled analysis of 7 studies (248 vs. 251 participants) gave SMD 0.90 with I² = 92.5%, indicating substantial heterogeneity.
- An exploratory subgroup analysis suggested larger effects in practice courses than theory courses, but only 2 practice-oriented studies drove it.
- GRADE-informed certainty ranged from very low to low.
What the review found when it combined the studies
These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.
Strict knowledge acquisition of medical learners taught with LLM tools versus traditional teaching
SMD 0.80 (95% CI 0.07 to 1.53)
5 studies · search to 31 March 2026
The review’s own words
The strict knowledge acquisition sensitivity analysis included 5 studies (164 vs. 164 participants; Fig. 4 ) and yielded a pooled effect of SMD 0.80 (95% CI 0.07 to 1.53; I² = 88.6%).
Delayed retention of knowledge in medical learners taught with LLM tools versus traditional teaching
SMD 0.15 (95% CI -0.50 to 0.80)
2 studies · search to 31 March 2026
The review’s own words
Delayed retention included 2 studies (66 vs. 82 participants; Fig. 5 ), with a pooled effect of SMD 0.15 (95% CI -0.50 to 0.80).
Post-course scores in practice courses with LLM-based teaching versus non-AI teaching
SMD 1.60 (95% CI -0.21 to 3.41)
search to 31 March 2026
The review’s own words
Exploratory subgroup analysis showed a pooled effect of SMD 0.60 (95% CI -0.10 to 1.30) for theory courses and SMD 1.60 (95% CI -0.21 to 3.41) for practice courses.
Post-course score-based outcomes of medical learners taught with LLM tools versus non-AI teaching
SMD 0.90 (95% CI 0.18 to 1.62)
search to 31 March 2026
The review’s own words
The pooled analysis suggested that LLM-based teaching was associated with higher post-course score-based outcomes than non-AI control teaching, with a pooled effect size of SMD 0.90 (95% CI 0.18 to 1.62; I² = 92.5%; P = 0.015).
Post-course scores in theory courses with LLM-based teaching versus non-AI teaching
SMD 0.60 (95% CI -0.10 to 1.30)
search to 31 March 2026
The review’s own words
Exploratory subgroup analysis showed a pooled effect of SMD 0.60 (95% CI -0.10 to 1.30) for theory courses and SMD 1.60 (95% CI -0.21 to 3.41) for practice courses.
Structured clinical performance of medical learners taught with LLM tools versus traditional teaching
SMD 3.39 (95% CI 1.43 to 5.35)
2 studies · 77 people · search to 31 March 2026
The review’s own words
Structured clinical performance included 2 studies (38 vs. 39 participants), with a pooled effect of SMD 3.39 (95% CI 1.43 to 5.35) (Fig. 6 ).
Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results
The full story
The review was not prospectively registered and no public protocol was available; it grew out of a broader evidence-mapping exercise. Risk of bias was appraised with RoB 2, ROBINS-I or JBI-informed domains, and pooled analyses used an inverse-variance random-effects model.
Of the 14 included studies, seven did not enter the pooled score-based synthesis directly. One was kept in the structured clinical performance supplementary meta-analysis, and six were shown through classification summary figures and single-study exploratory displays.
The authors conclude that some LLM-based teaching interventions may improve structured clinical performance and certain short-term knowledge outcomes, while delayed retention remains uncertain. They say the broader pooled score should not be read as a single universal endpoint.
Sources
1 on file- BMC Medical Educationdoi:10.1186/s12909-026-09879-0
Talk it over
Who gets this benefit, and who was left out of the study or story?
Review of 14 studies finds AI teaching tools may lift medical students' knowledge scoreshttps://truvace.com/item/effectiveness-of-large-language-model-based-teaching-interventions-in-medical-ed
Who gets this benefit, and who was left out of the study or story?
The gain A systematic review of 14 studies with 1,207 medical learners found AI chatbot-based teaching was linked to higher knowledge scores (SMD 0.80, 95% CI 0.07 to 1.53, 5 studies) than traditional teaching, though certainty was low.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a systematic review or meta-analysis with a sample of 1,207, from BMC Medical Education.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat Pooled domains rested on only 2 to 5 studies, certainty was low to very low, and the clinical performance estimate may be inflated by small-study effects, selective reporting or measurement artifacts.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say