Truace
Subscribe
70/100

Moderate evidence

Trace: a gain and a problemPublished recordHealthThe Trace

Full story
More
Export record JSONEmailLinkedInX

Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis

Truvace record
TRV-2026-2853raw data (JSON)
Version
v1Published
First published
Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
sha256
ab43d20ef580…check it

BACKGROUND: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. OBJECTIVE: In this study, we reviewed all studies on Ch…

63Gain side

How the two sides scoreClose call

67Problem side

The gain

A systematic review of 45 studies found GPT-4 had a pooled accuracy of 81% on medical licensing exam questions versus 58% for GPT-3.5, passing 26 of 29 exams and beating average medical-student scores in 13 of 17 comparisons.

The problem

The same 45-study review found GPT-4 scored only 52% on the Korean medical licensing exam, accuracy on image-based questions ranged from 13% to 100%, and ChatGPT did significantly worse on open-ended questions than multiple-choice ones; the authors judged it not yet suitable for medical education.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

What this doesn’t fix

The review excluded specialty, dental, pharmacy and other medical exams as well as non-English publications, so its findings cover only comprehensive national licensing exams reported in English.

Sources

1 on file
  1. Journal of Medical Internet Researchdoi:10.2196/60807
    Peer-reviewed2024-06-15

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • GPT-3.5 accuracy on licensing examinations when task prompts were given

    Accuracy 68% (95% CI 57% to 77%)

    search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 in examinations with prompts was 68% (95% CI 57-77; P <.01), and in examinations without prompts, it was 54% (95% CI 50-59; P <.01).

  • GPT-4 accuracy on licensing examinations from English-speaking countries

    Accuracy 86% (95% CI 82% to 89%)

    search to March 29, 2024

    The review’s own words
    For GPT-4, the integrated accuracy in examinations from English-speaking countries was 86% (95% CI 82%-89%; P <.01), and in examinations from non–English-speaking countries with original languages, it was 80% (95% CI 76-83; P <.01).

  • Accuracy of GPT-4 on medical licensing examination questions

    Accuracy 81% (95% CI 78% to 84%)

    29 studies · search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 was 58% (95% CI 53-63; P <.01), and the integrated accuracy for GPT-4 was 81% (95% CI 78-84; P <.01; Figures S1 and S2 in Multimedia Appendix 4 ).

  • Accuracy of GPT-3.5 on medical licensing examination questions

    Accuracy 58% (95% CI 53% to 63%)

    25 studies · search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-3.5 was 58% (95% CI 53-63; P <.01), and the integrated accuracy for GPT-4 was 81% (95% CI 78-84; P <.01; Figures S1 and S2 in Multimedia Appendix 4 ).

  • GPT-4 accuracy on non-English-speaking countries' examinations in original language

    Accuracy 80% (95% CI 76% to 83%)

    search to March 29, 2024

    The review’s own words
    For GPT-4, the integrated accuracy in examinations from English-speaking countries was 86% (95% CI 82%-89%; P <.01), and in examinations from non–English-speaking countries with original languages, it was 80% (95% CI 76-83; P <.01).

  • GPT-4 accuracy on licensing examinations when task prompts were given

    Accuracy 85% (95% CI 83% to 88%)

    search to March 29, 2024

    The review’s own words
    The integrated accuracy for GPT-4 in examinations with prompts was 85% (95% CI 83-88; P <.01), and in examinations without prompts, it was 79% (95% CI 75-82; P <.01).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

Cite this record

v1

Truvace Impact Record TRV-2026-2853, v1: “Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis.” Truvace, 2026-11-12. /record/TRV-2026-2853 (accessed on the day you copy it). sha256 ab43d20ef580e8b8…

Change history

1 version

Every change made to this record since it was first published, and why. Nothing has changed since it went up.

  1. Publishedv1ab43d20ef580…

    Certified into the record

Check this record hasn’t been altered

Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2853 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.