aceModerate evidence
Published recordEducationThe Trace
More
The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis
- Truvace record
- TRV-2026-2821raw data (JSON)
- Version
- v1Published
- First published
- Evidence type
- Systematic review or meta-analysis
- Sources
- 1 sourcePeer-reviewed
- sha256
- 2c4518babe81…check it
PURPOSE: Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans. METHODS: This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and…
56Gain side
How the two sides scoreClose call
56Problem side
The gain
In a network meta-analysis of studies comparing AI and human question writers, ChatGPT 4 produced multiple-choice questions rated similar to human-written ones for relevance, clarity and distractor quality, though the evidence was graded very low certainty.
The problem
In the same network meta-analysis, Llama 2 questions rated worse than human-written ones for clarity (mean difference -1.21 on a 10-point scale) and distractor quality (-1.50), and ChatGPT 3.5 did worse than ChatGPT 4 and Llama 2 on both.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
What this doesn’t fix
All main comparisons were rated very low certainty on GRADE, and the authors say further studies are needed that directly compare these models with human-generated questions and give the questions to students.
Sources
1 on file- PLOS Onedoi:10.1371/journal.pone.0340277
What the review found when it combined the studies
These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.
Item discrimination index of ChatGPT 3.5 questions compared with human questions
Mean difference −0.07 (95% CI −0.34 to 0.20)
search to November 1, 2024
The review’s own words
Pairwise analysis between ChatGPT 3.5 and humans demonstrated that there is no significant difference in the average item discrimination index (MD −0.07; 95% CI: −0.34, 0.20; GRADE: VERY LOW) (Fig 7a).
Relevance of multiple-choice questions generated by Llama 2 compared with humans
Mean difference −0.76 (95% CI −1.27 to −0.25)
search to November 1, 2024
The review’s own words
but MCQs generated by Llama 2 may be less relevant than those created by humans (MD −0.76; 95% CI: −1.27, −0.25; GRADE: VERY LOW)
Difficulty index of multiple-choice questions from ChatGPT 3.5 compared with humans
Mean difference 5.86 (95% CI −8.49 to 20.20)
search to November 1, 2024
The review’s own words
Pairwise analysis between studies comparing ChatGPT 3.5 and humans demonstrated that there is no significant difference in the difficulty index (MD 5.86; 95% CI: −8.49, 20.20; GRADE: VERY LOW) (Fig 6a).
Point biserial correlation of ChatGPT 3.5 questions compared with human questions
Mean difference −0.05 (95% CI −0.21 to 0.11)
search to November 1, 2024
The review’s own words
Pairwise meta-analysis demonstrated no significant difference in scores between the Chat GPT 3.5 and human generated questions (MD: −0.05; 95% CI: −0.21 to 0.11; GRADE: VERY LOW) (Fig 8a).
Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results
Cite this record
v1Truvace Impact Record TRV-2026-2821, v1: “The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis.” Truvace, 2026-11-12. /record/TRV-2026-2821 (accessed on the day you copy it). sha256 2c4518babe81e5f3…
Change history
1 versionEvery change made to this record since it was first published, and why. Nothing has changed since it went up.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2821 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.