Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 45 studies finds GPT-4 passed most medical licensing exams, but not all

Study title: Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis

Evidence type
Systematic review or meta-analysis
Sample
n = 45
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2853cite or check it

Study title: Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis

A systematic review and meta-analysis in the Journal of Medical Internet Research pooled 45 studies, published between January 2022 and March 2024, that tested ChatGPT on national medical licensing exams from 17 countries and regions. GPT-4 reached a pooled accuracy of 81% on multiple-choice questions, against 58% for GPT-3.5.

Read the study abstract

BACKGROUND: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. OBJECTIVE: In this study, we reviewed all studies on Ch…

63Gain side

How the two sides scoreClose call

67Problem side

The gain

A systematic review of 45 studies found GPT-4 had a pooled accuracy of 81% on medical licensing exam questions versus 58% for GPT-3.5, passing 26 of 29 exams and beating average medical-student scores in 13 of 17 comparisons.

The problem

The same 45-study review found GPT-4 scored only 52% on the Korean medical licensing exam, accuracy on image-based questions ranged from 13% to 100%, and ChatGPT did significantly worse on open-ended questions than multiple-choice ones; the authors judged it not yet suitable for medical education.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

GPT-4 passed the exam in 26 of 29 cases and beat average medical-student scores in 13 of 17 comparisons. GPT-3.5 passed in 10 of 26 studies and beat students in 4 of 14. Results varied widely, though: GPT-4 managed only 52% on the Korean exam, and its accuracy on image-based questions ranged from 13% to 100%.

The authors conclude that GPT-4 shows considerable potential for medical education but is not yet suitable for it, citing insufficient accuracy, inconsistent performance and differences in medical policy and knowledge between countries. They also note that ChatGPT can give detailed, convincing explanations for wrong answers.

Main points

  1. GPT-4's pooled accuracy was 81% (95% CI 78-84) across 29 studies, versus 58% (95% CI 53-63) for GPT-3.5 across 25 studies.
  2. GPT-4 passed in 26 of 29 studies (89.7%); GPT-3.5 passed or performed satisfactorily in 10 of 26 (38.5%).
  3. GPT-4 surpassed average medical-student scores in 13 of 17 comparisons (76.5%); GPT-3.5 did so in 4 of 14 (28.6%).
  4. Translating questions into English helped GPT-3.5, by more than 20% in two studies, but changed GPT-4's accuracy by only 0.17% to 8.65%.
  5. GPT-4 scored 86% on exams from English-speaking countries versus 80% in original-language exams elsewhere (P=.02); GPT-3.5 showed no gap (57% vs 58%).
  6. Prompts raised accuracy: GPT-3.5 from 54% to 68% and GPT-4 from 79% to 85%.
  7. On image-based questions, a Japanese study found GPT-4 scored 72% (78/108) with text only versus 68% (73/108) with images and text, and a Chilean study found GPT-4V (70%) below GPT-4 (76.7%).
  8. The authors say ChatGPT scored worse on difficult questions and, in two of four studies, on open-ended questions than on multiple-choice ones.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • GPT-3.5 accuracy on licensing examinations when task prompts were given

    Accuracy 68% (95% CI 57% to 77%)

    search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 in examinations with prompts was 68% (95% CI 57-77; P <.01), and in examinations without prompts, it was 54% (95% CI 50-59; P <.01).

  • GPT-4 accuracy on licensing examinations from English-speaking countries

    Accuracy 86% (95% CI 82% to 89%)

    search to March 29, 2024

    The review’s own words
    For GPT-4, the integrated accuracy in examinations from English-speaking countries was 86% (95% CI 82%-89%; P <.01), and in examinations from non–English-speaking countries with original languages, it was 80% (95% CI 76-83; P <.01).

  • Accuracy of GPT-4 on medical licensing examination questions

    Accuracy 81% (95% CI 78% to 84%)

    29 studies · search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 was 58% (95% CI 53-63; P <.01), and the integrated accuracy for GPT-4 was 81% (95% CI 78-84; P <.01; Figures S1 and S2 in Multimedia Appendix 4 ).

  • Accuracy of GPT-3.5 on medical licensing examination questions

    Accuracy 58% (95% CI 53% to 63%)

    25 studies · search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 was 58% (95% CI 53-63; P <.01), and the integrated accuracy for GPT-4 was 81% (95% CI 78-84; P <.01; Figures S1 and S2 in Multimedia Appendix 4 ).

  • GPT-4 accuracy on non-English-speaking countries' examinations in original language

    Accuracy 80% (95% CI 76% to 83%)

    search to March 29, 2024

    The review’s own words
    For GPT-4, the integrated accuracy in examinations from English-speaking countries was 86% (95% CI 82%-89%; P <.01), and in examinations from non–English-speaking countries with original languages, it was 80% (95% CI 76-83; P <.01).

  • GPT-4 accuracy on licensing examinations when task prompts were given

    Accuracy 85% (95% CI 83% to 88%)

    search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-4 in examinations with prompts was 85% (95% CI 83-88; P <.01), and in examinations without prompts, it was 79% (95% CI 75-82; P <.01).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors searched Web of Science, PubMed and Scopus for studies from January 1, 2022 to March 29, 2024, and added two preprints from Google Scholar. After removing duplicates and screening 8,610 abstracts, 45 papers met the criteria. Two reviewers screened and extracted data independently, using a modified QUADAS-2 framework to assess quality. The review was registered in PROSPERO.

The exams came from 17 countries and regions, with the United States (7), China (7) and Japan (6) most common. Of the 45 studies, 29 tested GPT-4 and 26 tested GPT-3.5. Eleven used exams from English-speaking countries. Only 2 studies were rated high risk in the index-test category and 3 in flow and timing. Heterogeneity was very high (I² of 95% for GPT-3.5 and 93% for GPT-4), so random-effects models were used.

The authors argue that exam passes do not make ChatGPT a reliable teacher. Only 2 GPT-4 results exceeded 90% accuracy, and the single perfect score came from a UK test of just 20 questions. GPT-4 also failed one Japanese exam with 67% and picked euthanasia on a Japanese question where it is illegal, which the authors cite as a local-policy problem.

They also say local traditional medicine, such as Chinese medicine, may be poorly covered in ChatGPT's training data. They warn that ChatGPT can give logical-sounding explanations for wrong answers and that students may memorise them. No publication bias was detected, and excluding single studies did not change the pooled results.

What this doesn’t fix

The review excluded specialty, dental, pharmacy and other medical exams as well as non-English publications, so its findings cover only comprehensive national licensing exams reported in English.

Sources

1 on file
  1. Journal of Medical Internet Researchdoi:10.2196/60807
    Peer-reviewed2024-06-15

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Review of 45 studies finds GPT-4 passed most medical licensing exams, but not allhttps://truvace.com/item/performance-of-chatgpt-across-different-versions-in-medical-licensing-examinatio

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain A systematic review of 45 studies found GPT-4 had a pooled accuracy of 81% on medical licensing exam questions versus 58% for GPT-3.5, passing 26 of 29 exams and beating average medical-student scores in 13 of 17 comparisons.

    The problem The same 45-study review found GPT-4 scored only 52% on the Korean medical licensing exam, accuracy on image-based questions ranged from 13% to 100%, and ChatGPT did significantly worse on open-ended questions than multiple-choice ones; the authors judged it not yet suitable for medical education.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 45, from Journal of Medical Internet Research.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The review excluded specialty, dental, pharmacy and other medical exams as well as non-English publications, so its findings cover only comprehensive national licensing exams reported in English.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health