aceModerate evidence
Published recordEducationGood Space
More
Can large language models generate exam questions comparable to humans? A systematic review and meta-analysis study in medical education
- Truvace record
- TRV-2026-2779raw data (JSON)
- Version
- v1Published
- First published
- Evidence type
- Systematic review or meta-analysis
- Sources
- 1 sourcePeer-reviewed
- sha256
- c5aadb5256a9…check it
Introduction Recently developed artificial intelligence (AI), particularly large language models (LLMs), e.g. GPT, streamline the development of multiple-choice questions (MCQs); however, concerns remain about psychometric quality, fairness across learner groups, and potential bias.Methods Following PRISMA 2020, we searched PubMed, Google Scholar, Web of Science, and the Cochrane Library for studies comparing both approaches in health education. We extracted difficulty and discrimination outcomes and assessed ri…
The gain
In a meta-analysis of 12 studies in health education, AI-generated multiple-choice questions showed no difference from human-written ones on difficulty (SMD 0.05) or discrimination (SMD −0.10).
What this doesn’t fix
The evidence base is limited, with little attention to equity, and between-study heterogeneity was high; excluding one study altered the discrimination result.
Sources
1 on file- Medical Teacherdoi:10.1080/0142159x.2026.2691072
What the review found when it combined the studies
These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.
Difficulty of multiple-choice exam items, AI-generated versus human-authored, in health education
SMD 0.05 (95% CI -0.30 to 0.40)
The review’s own words
There was no evidence of a difference between AI-generated and human-authored items for difficulty (SMD 0.05, 95% CI -0.30 to 0.40; p = 0.77; I2 = 82%) or discrimination (SMD -0.10, 95% CI -0.35 to 0.15; p = 0.42; I2 = 70%).
Discrimination of multiple-choice exam items, AI-generated versus human-authored, in health education
SMD -0.10 (95% CI -0.35 to 0.15)
The review’s own words
There was no evidence of a difference between AI-generated and human-authored items for difficulty (SMD 0.05, 95% CI -0.30 to 0.40; p = 0.77; I2 = 82%) or discrimination (SMD -0.10, 95% CI -0.35 to 0.15; p = 0.42; I2 = 70%).
Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results
Cite this record
v1Truvace Impact Record TRV-2026-2779, v1: “Can large language models generate exam questions comparable to humans? A systematic review and meta-analysis study in medical education.” Truvace, 2026-11-12. /record/TRV-2026-2779 (accessed on the day you copy it). sha256 c5aadb5256a90242…
Change history
1 versionEvery change made to this record since it was first published, and why. Nothing has changed since it went up.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2779 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.