The exact text we saved

This is the version exactly as it was saved, character for character. The SHA-256 hash below is a 64-character code worked out from that text. Run the same text through SHA-256 and you get the same code; change a single letter and the code comes out completely different. Each version also stores the previous version’s hash, which chains them together so no version can be swapped out quietly.

TRUVACE RECORD VERSION
record: TRV-2026-2853
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-12T17:35:10.888403Z
status: published
lens: trace
sector: health
headline: Performance of ChatGPT Across Different Versions in Medical Licensing Examinations Worldwide: Systematic Review and Meta-Analysis
dek: BACKGROUND: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations. OBJECTIVE: In this study, we reviewed all studies on Ch…
gain_title: A systematic review of 45 studies found GPT-4 had a pooled accuracy of 81% on medical licensing exam questions versus 58% for GPT-3.5, passing 26 of 29 exams and beating average medical-student scores in 13 of 17 comparisons.
problem_title: The same 45-study review found GPT-4 scored only 52% on the Korean medical licensing exam, accuracy on image-based questions ranged from 13% to 100%, and ChatGPT did significantly worse on open-ended questions than multiple-choice ones; the authors judged it not yet suitable for medical education.
trace_subject: ChatGPT performance on national medical licensing examinations
gain_reading: A systematic review of 45 studies found GPT-4 had a pooled accuracy of 81% on medical licensing exam questions versus 58% for GPT-3.5, passing 26 of 29 exams and beating average medical-student scores in 13 of 17 comparisons.
gain_evidence: GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. | GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5.
problem_reading: The same 45-study review found GPT-4 scored only 52% on the Korean medical licensing exam, accuracy on image-based questions ranged from 13% to 100%, and ChatGPT did significantly worse on open-ended questions than multiple-choice ones; the authors judged it not yet suitable for medical education.
problem_evidence: it only scored 52% on the Korean medical licensing examination, even lower than the overall accuracy of GPT-3.5 (58%) | ChatGPT performed significantly worse on open-ended questions than on MCQs. | GPT-4 is not yet suitable for use in medical education.
quick_read: A systematic review and meta-analysis in the Journal of Medical Internet Research pooled 45 studies, published between January 2022 and March 2024, that tested ChatGPT on national medical licensing exams from 17 countries and regions. GPT-4 reached a pooled accuracy of 81% on multiple-choice questions, against 58% for GPT-3.5.

GPT-4 passed the exam in 26 of 29 cases and beat average medical-student scores in 13 of 17 comparisons. GPT-3.5 passed in 10 of 26 studies and beat students in 4 of 14. Results varied widely, though: GPT-4 managed only 52% on the Korean exam, and its accuracy on image-based questions ranged from 13% to 100%.

The authors conclude that GPT-4 shows considerable potential for medical education but is not yet suitable for it, citing insufficient accuracy, inconsistent performance and differences in medical policy and knowledge between countries. They also note that ChatGPT can give detailed, convincing explanations for wrong answers.
limitation: The review excluded specialty, dental, pharmacy and other medical exams as well as non-English publications, so its findings cover only comprehensive national licensing exams reported in English.
tag: Dual reading
key_points: GPT-4's pooled accuracy was 81% (95% CI 78-84) across 29 studies, versus 58% (95% CI 53-63) for GPT-3.5 across 25 studies. | GPT-4 passed in 26 of 29 studies (89.7%); GPT-3.5 passed or performed satisfactorily in 10 of 26 (38.5%). | GPT-4 surpassed average medical-student scores in 13 of 17 comparisons (76.5%); GPT-3.5 did so in 4 of 14 (28.6%). | Translating questions into English helped GPT-3.5, by more than 20% in two studies, but changed GPT-4's accuracy by only 0.17% to 8.65%. | GPT-4 scored 86% on exams from English-speaking countries versus 80% in original-language exams elsewhere (P=.02); GPT-3.5 showed no gap (57% vs 58%). | Prompts raised accuracy: GPT-3.5 from 54% to 68% and GPT-4 from 79% to 85%. | On image-based questions, a Japanese study found GPT-4 scored 72% (78/108) with text only versus 68% (73/108) with images and text, and a Chilean study found GPT-4V (70%) below GPT-4 (76.7%). | The authors say ChatGPT scored worse on difficult questions and, in two of four studies, on open-ended questions than on multiple-choice ones.
rundown: The authors searched Web of Science, PubMed and Scopus for studies from January 1, 2022 to March 29, 2024, and added two preprints from Google Scholar. After removing duplicates and screening 8,610 abstracts, 45 papers met the criteria. Two reviewers screened and extracted data independently, using a modified QUADAS-2 framework to assess quality. The review was registered in PROSPERO.

The exams came from 17 countries and regions, with the United States (7), China (7) and Japan (6) most common. Of the 45 studies, 29 tested GPT-4 and 26 tested GPT-3.5. Eleven used exams from English-speaking countries. Only 2 studies were rated high risk in the index-test category and 3 in flow and timing. Heterogeneity was very high (I² of 95% for GPT-3.5 and 93% for GPT-4), so random-effects models were used.

The authors argue that exam passes do not make ChatGPT a reliable teacher. Only 2 GPT-4 results exceeded 90% accuracy, and the single perfect score came from a UK test of just 20 questions. GPT-4 also failed one Japanese exam with 67% and picked euthanasia on a Japanese question where it is illegal, which the authors cite as a local-policy problem.

They also say local traditional medicine, such as Chinese medicine, may be poorly covered in ChatGPT's training data. They warn that ChatGPT can give logical-sounding explanations for wrong answers and that students may memorise them. No publication bias was detected, and excluding single studies did not change the pooled results.
sources:
- peer_reviewed | Journal of Medical Internet Research | https://doi.org/10.2196/60807 | 2024-06-15
prev: 0000000000000000000000000000000000000000000000000000000000000000

SHA-256 hash

ab43d20ef580e8b8ceeea1994f1fc949a56932e7ec95c3561b8227bac0e4a64f

Previous version’s hash

0000000000000000000000000000000000000000000000000000000000000000