Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 19 studies finds ChatGPT averaged 61.1% on medical exam questions

Study title: Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis

Evidence type
Systematic review or meta-analysis
Sample
n = 19
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2852cite or check it

Study title: Performance of ChatGPT in medical examinations: A systematic review and a meta‐analysis

A systematic review and meta-analysis of 19 peer-reviewed studies found that ChatGPT version 3.5 answered a mean 61.1% of multiple-choice questions correctly across medical examinations (95% CI 56.1%–66.0%). The authors describe this as approximately a passing grade.

Read the study abstract

The use of ChatGPT, an artificial intelligence (AI) language model, has been described in various scientific and medical applications.1 With its human-like conversation capacity and large quantity of data, ChatGPT has the potential to become an important medical education tool.2 ChatGPT's performance in different medical knowledge examinations has been recently studied in various medical disciplines. However, reported rates of correct answers vary dramatically across different examinations and medical fields.3,…

The problem

A meta-analysis of 19 studies found ChatGPT 3.5 answered a mean 61.1% of medical multiple-choice exam questions correctly, roughly a passing grade and well below the 95% the authors say a reliable educational tool would need.

In brief

Scores varied widely, from 40% on a biomedical admission test to 100% on a diabetes knowledge questionnaire. The authors say this makes ChatGPT's use for exam preparation something to approach with caution, since medical education and test preparation form a large industrial market.

They say a correct response rate above 95% might make ChatGPT a reliable educational tool, and that it is unknown whether future versions will reach that level because the training data was not built with medical education in mind.

Main points

  1. Searches of PubMed, Scopus and Web of Science ran up to 2 June 2023 and yielded 19 included articles.
  2. Only peer-reviewed studies of English-language multiple-choice questions using ChatGPT version 3.5 were included.
  3. Plastic surgery, the United States Medical Licensing Examinations and anaesthesia each accounted for two studies (11% each).
  4. The median exam had 242 questions, ranging from 20 in medical physiology to 3705 in anaesthesia.
  5. Pooled ChatGPT accuracy was 61.1%, with individual exams ranging from 40% to 100%.
  6. The authors advise that preparing for exams with ChatGPT should be done with proper caution.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • ChatGPT (version 3.5) correct-answer rate on medical multiple-choice examinations

    Percentage 61.1% (95% CI 56.1% to 66.0%)

    19 studies · search to 2 June 2023

    The review’s own words
    The mean performance of ChatGPT was 61.1% (95% CI 56.1%–66.0%

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

Two reviewers (RM and GL) screened titles and abstracts independently, with a third reviewer (YB) resolving disagreements. Studies were excluded if they tested non-English prompts, formats other than multiple-choice such as open questions or frequently asked questions, or versions other than 3.5.

From each study the team extracted the medical field, number of questions, number correct and a 95% confidence interval for the performance rate, then analysed the data with MedCalc and OpenMeta[Analyst]. Question counts averaged 524.5 with a standard deviation of 847.2.

The authors conclude ChatGPT correctly answered the majority of questions but only at around a passing level. They suggest future meta-analyses should cover newer chatbot versions to update the picture for medical education.

Sources

1 on file
  1. BJOG: An International Journal of Obstetrics & Gynaecologydoi:10.1111/1471-0528.17641
    Peer-reviewed2023-08-21

Talk it over

Four questions to start onWho carries this cost, and what would have to change to prevent it?

Start with a question

Review of 19 studies finds ChatGPT averaged 61.1% on medical exam questionshttps://truvace.com/item/performance-of-chatgpt-in-medical-examinations-a-systematic-review-and-a-meta-an

  1. Who carries this cost, and what would have to change to prevent it?

    The problem A meta-analysis of 19 studies found ChatGPT 3.5 answered a mean 61.1% of medical multiple-choice exam questions correctly, roughly a passing grade and well below the 95% the authors say a reliable educational tool would need.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 19, from BJOG: An International Journal of Obstetrics & Gynaecology.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The review included only studies of ChatGPT version 3.5, the included studies were heterogeneous with the number of answer choices per exam unreported, and the authors say tool performance may be overestimated.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health