Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 43 studies finds medical students use ChatGPT with little school guidance

Study title: Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions

Evidence type
Systematic review or meta-analysis
Sample
n = 12,400
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2769cite or check it

Study title: Large language models in undergraduate medical education: a systematic review of examination performance, knowledge acquisition, and learner perceptions

A systematic review of 43 studies, covering roughly 12,400 participants in 19 countries, examined how ChatGPT-type language models perform on medical exams, affect learning and are viewed by undergraduate medical students. Among students, 82% reported getting no institutional guidance on using them.

Read the study abstract

BACKGROUND: Large language models (LLMs), particularly successive iterations of ChatGPT (GPT-3.5 and GPT-4), have demonstrated increasingly sophisticated performance on standardised medical assessments and are now widely accessible to undergraduate medical students worldwide. Although the evidence base has expanded rapidly since ChatGPT's public release in November 2022, findings remain fragmented across three largely distinct research domains: examination performance, knowledge acquisition, and learner percepti…

The problem

In a systematic review of 43 studies covering about 12,400 participants, 82% of medical students reported receiving no institutional guidance on using AI chatbots, and 61% raised academic integrity concerns.

In brief

GPT-4 passed USMLE Steps 1-3 in every study that tested it, with mean accuracy of 86% against 56% for GPT-3.5. Learning benefits were smaller: six of twelve studies found modest short-term gains, with no durable retention.

The authors call the lack of guidance the most urgent finding. Most of the underlying evidence is weak, with moderate-to-high risk of bias in 86% of the studies, so the learning results are tentative.

Main points

  1. The review screened 1,847 records and included 43 studies with about 12,400 participants across 19 countries, searching January 2022 to March 2026.
  2. GPT-4 passed USMLE Steps 1-3 in all studies that tested it (mean accuracy 86%, interquartile range 81% to 89%), well above GPT-3.5's mean of 56%.
  3. GPT-4 accuracy fell to 61%-71% on multi-step clinical vignettes and 48%-63% on image-dependent questions.
  4. Six of twelve studies reported short-term knowledge gains, with effect sizes of 0.31 to 0.71 (median about 0.53) and no durable retention.
  5. Of students surveyed, 74% viewed LLMs positively as study aids, yet 61% raised academic integrity concerns and 59% cited hallucination risk.
  6. Faculty were less positive than students, at 51% positive.

The full story

The review followed PRISMA 2020 and searched PubMed, Scopus and Web of Science, plus grey literature and citation tracking. Two reviewers independently screened records, extracted data and assessed bias with the Newcastle-Ottawa Scale and ROBINS-I. Results were synthesised narratively rather than pooled in a meta-analysis.

On exams, GPT-4 consistently reached passing thresholds, but the authors report clinically important weaknesses in multi-step reasoning, uncertainty calibration and tasks involving images. Accuracy on multi-step clinical vignettes was 61%-71% and on image-dependent items 48%-63%.

On learning and perceptions, students were broadly positive (74%), but 59% cited hallucination risk and 82% reported receiving no institutional guidance. The authors say longitudinal controlled research with validated outcome measures and standardised AI literacy curricula are the priorities.

Sources

1 on file
  1. BMC Medical Educationdoi:10.1186/s12909-026-09999-7
    Peer-reviewed2026-07-30

Talk it over

Four questions to start onWho carries this cost, and what would have to change to prevent it?

Start with a question

Review of 43 studies finds medical students use ChatGPT with little school guidancehttps://truvace.com/item/large-language-models-in-undergraduate-medical-education-a-systematic-review-of-

  1. Who carries this cost, and what would have to change to prevent it?

    The problem In a systematic review of 43 studies covering about 12,400 participants, 82% of medical students reported receiving no institutional guidance on using AI chatbots, and 61% raised academic integrity concerns.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 12,400, from BMC Medical Education.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Risk of bias was moderate-to-high in 86% of the included studies, and the learning gains were short-term only, without durable retention, so the benefit findings are tentative.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health