Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

ChatGPT 4 matched humans on exam question quality; Llama 2 fell short, review finds

Study title: The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2821cite or check it

Study title: The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis

A systematic review and network meta-analysis compared multiple-choice questions written by large language models with those written by humans for health professions education. ChatGPT 4 produced questions of similar quality to human authors on relevance, clarity and distractor quality, with mean differences close to zero.

Read the study abstract

PURPOSE: Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans. METHODS: This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and…

56Gain side

How the two sides scoreClose call

56Problem side

The gain

In a network meta-analysis of studies comparing AI and human question writers, ChatGPT 4 produced multiple-choice questions rated similar to human-written ones for relevance, clarity and distractor quality, though the evidence was graded very low certainty.

The problem

In the same network meta-analysis, Llama 2 questions rated worse than human-written ones for clarity (mean difference -1.21 on a 10-point scale) and distractor quality (-1.50), and ChatGPT 3.5 did worse than ChatGPT 4 and Llama 2 on both.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Results varied sharply by model. Llama 2 questions scored 1.21 points lower on clarity and 1.50 points lower on distractor quality on a 10-point scale, and exploratory tests found ChatGPT 3.5 did worse than both Llama 2 and ChatGPT 4 on those measures (p < 0.001).

Every reported comparison was graded very low certainty. The authors say further studies should compare these models directly against human-written questions and give the questions to students.

Main points

  1. The review was registered with PROSPERO (CRD42025608775) and used a frequentist random-effects network meta-analysis or pairwise meta-analysis.
  2. Ovid MEDLINE, Ovid EMBASE and Scopus were searched from inception to November 1, 2024.
  3. Five large language models were included in the analysis.
  4. Question quality was judged on seven outcomes, including relevance, clarity, accuracy, distractor quality, item difficulty and item discrimination.
  5. Continuous data were converted to a 10-point scale and reported as mean differences.
  6. ChatGPT 4 versus humans: relevance MD -0.13, clarity MD -0.03, distractor quality MD -0.10, with confidence intervals spanning zero.
  7. Llama 2 versus humans: clarity MD -1.21 and distractor quality MD -1.50, with intervals excluding zero.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Item discrimination index of ChatGPT 3.5 questions compared with human questions

    Mean difference −0.07 (95% CI −0.34 to 0.20)

    search to November 1, 2024

    The review’s own words
    Pairwise analysis between ChatGPT 3.5 and humans demonstrated that there is no significant difference in the average item discrimination index (MD −0.07; 95% CI: −0.34, 0.20; GRADE: VERY LOW) (Fig 7a).

  • Relevance of multiple-choice questions generated by Llama 2 compared with humans

    Mean difference −0.76 (95% CI −1.27 to −0.25)

    search to November 1, 2024

    The review’s own words
    but MCQs generated by Llama 2 may be less relevant than those created by humans (MD −0.76; 95% CI: −1.27, −0.25; GRADE: VERY LOW)

  • Difficulty index of multiple-choice questions from ChatGPT 3.5 compared with humans

    Mean difference 5.86 (95% CI −8.49 to 20.20)

    search to November 1, 2024

    The review’s own words
    Pairwise analysis between studies comparing ChatGPT 3.5 and humans demonstrated that there is no significant difference in the difficulty index (MD 5.86; 95% CI: −8.49, 20.20; GRADE: VERY LOW) (Fig 6a).

  • Point biserial correlation of ChatGPT 3.5 questions compared with human questions

    Mean difference −0.05 (95% CI −0.21 to 0.11)

    search to November 1, 2024

    The review’s own words
    Pairwise meta-analysis demonstrated no significant difference in scores between the Chat GPT 3.5 and human generated questions (MD: −0.05; 95% CI: −0.21 to 0.11; GRADE: VERY LOW) (Fig 8a).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors searched three databases through November 1, 2024, and pooled studies in a network meta-analysis so that several models could be ranked against human question writers. Risk of bias was assessed with the MERSQI tool and certainty of evidence with GRADE network meta-analysis guidelines.

Quality was measured on seven predefined outcomes: question relevance, clarity and accuracy; distractor quality; item difficulty; and two item discrimination measures (point biserial correlation and discrimination index). Continuous results were put on a 10-point scale for comparison.

ChatGPT 4 landed within a fraction of a point of human authors on relevance, clarity and distractors, while Llama 2 trailed by more than a point on clarity and distractors. Post-hoc t-tests also placed ChatGPT 3.5 behind ChatGPT 4 and Llama 2 on those two measures. The authors conclude that ChatGPT 4 may create similar quality questions to humans, whereas ChatGPT 3.5 and Llama 2 may be worse.

What this doesn’t fix

All main comparisons were rated very low certainty on GRADE, and the authors say further studies are needed that directly compare these models with human-generated questions and give the questions to students.

Sources

1 on file
  1. PLOS Onedoi:10.1371/journal.pone.0340277
    Peer-reviewed2026-01-02

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

ChatGPT 4 matched humans on exam question quality; Llama 2 fell short, review findshttps://truvace.com/item/the-use-of-large-language-models-in-generating-multiple-choice-questions-for-hea

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a network meta-analysis of studies comparing AI and human question writers, ChatGPT 4 produced multiple-choice questions rated similar to human-written ones for relevance, clarity and distractor quality, though the evidence was graded very low certainty.

    The problem In the same network meta-analysis, Llama 2 questions rated worse than human-written ones for clarity (mean difference -1.21 on a 10-point scale) and distractor quality (-1.50), and ChatGPT 3.5 did worse than ChatGPT 4 and Llama 2 on both.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from PLOS One.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat All main comparisons were rated very low certainty on GRADE, and the authors say further studies are needed that directly compare these models with human-generated questions and give the questions to students.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education