70/100

Moderate evidence

Cite
More
EmailLinkedInX

Review of 54 studies shows AI-written exam questions were easier, with weaker wrong answers

Study title: Large language model-generated vs. human-authored assessment items in education: systematic review and meta-analysis with a focus on health professional education

Evidence type
Systematic review or meta-analysis
Sample
n = 54
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Large language model-generated vs. human-authored assessment items in education: systematic review and meta-analysis with a focus on health professional education

A systematic review and meta-analysis pooled 54 studies that used large language models to write assessment items and reported psychometric results, mostly in health professional education. In direct comparisons with human-written items, the AI items were significantly easier and more often contained non-functioning distractors, meaning wrong answer options that test-takers rarely pick.

Read the study abstract

Purpose Assessment is crucial in health professional education (HPE) but is resource-intensive to create. Large language models (LLMs) offer a potential solution by generating assessment items; however, a comprehensive synthesis of their psychometric quality is lacking in HPE. Design/methodology/approach We systematically searched the electronic databases and included studies generating assessment items using any LLM and reporting psychometric outcomes. The primary outcomes were item difficulty index, discrimina…

The problem

A systematic review and meta-analysis of 54 studies found AI-generated assessment items were easier than human-written ones (difficulty index g 0.40) and had higher odds of containing non-functioning distractors (OR 1.86).

In brief

The pooled difficulty index was higher for AI items (g 0.40) and the odds of non-functioning distractors were 1.86 times those of human items. Discrimination index, which shows how well an item separates strong from weak students, did not differ significantly (g -0.21), but distractor effectiveness was poorer (g -0.65).

When AI items were assessed on their own, 67% were accurate or needed only minor revisions, yet 37% of distractors were non-functioning. Rankings across all main psychometric measures placed human-authored items first, and the authors say expert review and careful prompting are needed to produce valid, reliable tests.

Main points

  1. The review included 54 studies, and 92.8% of those with direct comparisons were in health professional education.
  2. AI-generated items had a higher difficulty index than human items (pooled g 0.40, 95% CI 0.23 to 0.57), meaning they were easier.
  3. The odds of containing non-functioning distractors were 1.86 times higher for AI items (95% CI 1.27 to 2.73).
  4. Discrimination index showed no significant difference (g -0.21, 95% CI -0.49 to 0.07), while distractor effectiveness was poorer for AI items (g -0.65).
  5. Assessed alone, 67% of AI items were accurate or needed only minor edits, but 37% of their distractors were non-functioning.
  6. Surface under the cumulative ranking curve analyses ranked human-generated items superior on every critical psychometric parameter.

The full story

Researchers searched electronic databases for studies in which any large language model generated assessment items and psychometric outcomes were reported. The primary outcomes were item difficulty index, discrimination index, distractor effectiveness and the risk of non-functioning distractors, pooled with random-effects models.

In head-to-head comparisons, AI items were easier and had more flawed answer options than human-written ones, though they separated strong from weak test-takers about as well. Ranking analyses consistently favoured human-authored items.

The authors conclude that AI can produce assessment items efficiently and that many are usable with minor edits, but that sophisticated prompt engineering and expert-led review are essential for valid and reliable assessments in health professional education.

Sources

1 on file
  1. Peer-reviewed2026-11-18

Talk it over

Four questions to start on
Who carries this cost, and what would have to change to prevent it?

Start with a question

Review of 54 studies shows AI-written exam questions were easier, with weaker wrong answershttps://truvace.com/item/large-language-model-generated-vs-human-authored-assessment-items-in-education-s

  1. Who carries this cost, and what would have to change to prevent it?

    The problem A systematic review and meta-analysis of 54 studies found AI-generated assessment items were easier than human-written ones (difficulty index g 0.40) and had higher odds of containing non-functioning distractors (OR 1.86).

    Answer in the thread
  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 54, from Journal of Applied Research in Higher Education.

    Answer in the thread
  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The certainty of the evidence ranged between very low and moderate, so the pooled differences between AI and human items carry real uncertainty.

    Answer in the thread
  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say