aceModerate evidence
More
AI-written medical exam questions matched human ones in a review of 12 studies
Study title: Can large language models generate exam questions comparable to humans? A systematic review and meta-analysis study in medical education
- Evidence type
- Systematic review or meta-analysis
- Sample
- n = 12
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2779cite or check it
Study title: Can large language models generate exam questions comparable to humans? A systematic review and meta-analysis study in medical education
A systematic review and meta-analysis pooled 12 studies comparing multiple-choice questions written by large language models with those written by humans in health education. It found no difference in difficulty (SMD 0.05, 95% CI −0.30 to 0.40) or in discrimination (SMD −0.10, 95% CI −0.35 to 0.15).
Read the study abstract
Introduction Recently developed artificial intelligence (AI), particularly large language models (LLMs), e.g. GPT, streamline the development of multiple-choice questions (MCQs); however, concerns remain about psychometric quality, fairness across learner groups, and potential bias.Methods Following PRISMA 2020, we searched PubMed, Google Scholar, Web of Science, and the Cochrane Library for studies comparing both approaches in health education. We extracted difficulty and discrimination outcomes and assessed ri…
The gain
In a meta-analysis of 12 studies in health education, AI-generated multiple-choice questions showed no difference from human-written ones on difficulty (SMD 0.05) or discrimination (SMD −0.10).
In brief
The authors say this makes AI-generated items best suited to supplementing expert item writing in lower-stakes settings. They propose staged, auditable safety checkpoints, including provenance tracking, continuous psychometric monitoring and human gatekeeping.
Results varied widely between studies, and the evidence base is limited, with little attention to equity. Human-written questions were more difficult in medical licensing examinations, and dropping a single study changed the discrimination result to favour AI.
Main points
- Twelve studies were included: two randomised trials and ten non-randomised studies.
- Eleven studies supplied learner-derived difficulty data and twelve supplied discrimination data.
- Difficulty showed no pooled difference (SMD 0.05, p = 0.77, I2 = 82%).
- Discrimination showed no pooled difference (SMD −0.10, p = 0.42, I2 = 70%).
- Human-generated questions were more difficult in medical licensing examinations.
- Removing one study in sensitivity analysis showed a significant benefit for AI-generated items on discrimination.
- The authors describe this as the first review comparing AI and human-generated MCQs.
What the review found when it combined the studies
These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.
Difficulty of multiple-choice exam items, AI-generated versus human-authored, in health education
SMD 0.05 (95% CI -0.30 to 0.40)
The review’s own words
There was no evidence of a difference between AI-generated and human-authored items for difficulty (SMD 0.05, 95% CI -0.30 to 0.40; p = 0.77; I2 = 82%) or discrimination (SMD -0.10, 95% CI -0.35 to 0.15; p = 0.42; I2 = 70%).
Discrimination of multiple-choice exam items, AI-generated versus human-authored, in health education
SMD -0.10 (95% CI -0.35 to 0.15)
The review’s own words
There was no evidence of a difference between AI-generated and human-authored items for difficulty (SMD 0.05, 95% CI -0.30 to 0.40; p = 0.77; I2 = 82%) or discrimination (SMD -0.10, 95% CI -0.35 to 0.15; p = 0.42; I2 = 70%).
Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results
The full story
The authors followed PRISMA 2020 and searched PubMed, Google Scholar, Web of Science and the Cochrane Library for studies comparing AI and human question writing in health education. Risk of bias was assessed with the original Cochrane tool for randomised trials and ROBINS-I for non-randomised studies, and random-effects meta-analyses were run.
Heterogeneity was high for both outcomes (I2 of 82% for difficulty and 70% for discrimination). Subgroup analyses by AI model and educational domain did not fully explain it.
The authors recommend a precautionary approach with provenance, continuous psychometric surveillance and human gatekeeping, to support equitable, culturally safe, high-quality outputs. They frame AI items as supplements to expert development rather than replacements for now.
Sources
1 on file- Medical Teacherdoi:10.1080/0142159x.2026.2691072
Talk it over
Who gets this benefit, and who was left out of the study or story?
AI-written medical exam questions matched human ones in a review of 12 studieshttps://truvace.com/item/can-large-language-models-generate-exam-questions-comparable-to-humans-a-systema
Who gets this benefit, and who was left out of the study or story?
The gain In a meta-analysis of 12 studies in health education, AI-generated multiple-choice questions showed no difference from human-written ones on difficulty (SMD 0.05) or discrimination (SMD −0.10).
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a systematic review or meta-analysis with a sample of 12, from Medical Teacher.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat The evidence base is limited, with little attention to equity, and between-study heterogeneity was high; excluding one study altered the discrimination result.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say