The exact text we saved

This is the version exactly as it was saved, character for character. The SHA-256 hash below is a 64-character code worked out from that text. Run the same text through SHA-256 and you get the same code; change a single letter and the code comes out completely different. Each version also stores the previous version’s hash, which chains them together so no version can be swapped out quietly.

TRUVACE RECORD VERSION
record: TRV-2026-2821
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-12T17:29:21.705522Z
status: published
lens: trace
sector: education
headline: The use of large language models in generating multiple choice questions for health professions education: A systematic review and network meta-analysis
dek: PURPOSE: Large language models (LLMs) have the potential to change medical education. Whether LLMs can generate multiple-choice questions (MCQs) that are of similar quality to those created by humans is unclear. This investigation assessed the quality of MCQs generated by LLMs compared to humans. METHODS: This review was registered with PROSPERO (CRD42025608775). A systematic review and frequentist random-effects network meta-analysis (NMA) or pairwise meta-analysis was performed. Ovid MEDLINE, Ovid EMBASE, and…
gain_title: In a network meta-analysis of studies comparing AI and human question writers, ChatGPT 4 produced multiple-choice questions rated similar to human-written ones for relevance, clarity and distractor quality, though the evidence was graded very low certainty.
problem_title: In the same network meta-analysis, Llama 2 questions rated worse than human-written ones for clarity (mean difference -1.21 on a 10-point scale) and distractor quality (-1.50), and ChatGPT 3.5 did worse than ChatGPT 4 and Llama 2 on both.
trace_subject: LLM-generated multiple-choice questions for health professions education
gain_reading: In a network meta-analysis of studies comparing AI and human question writers, ChatGPT 4 produced multiple-choice questions rated similar to human-written ones for relevance, clarity and distractor quality, though the evidence was graded very low certainty.
gain_evidence: ChatGPT 4 generated similar quality MCQs to humans with regards to question relevance (MD -0.13; 95% CI: -0.44,0.18; GRADE: VERY LOW) | distractor quality (MD -0.10; 95% CI: -0.24,0.04; GRADE: VERY LOW)
problem_reading: In the same network meta-analysis, Llama 2 questions rated worse than human-written ones for clarity (mean difference -1.21 on a 10-point scale) and distractor quality (-1.50), and ChatGPT 3.5 did worse than ChatGPT 4 and Llama 2 on both.
problem_evidence: MCQs generated by Llama 2 performed worse than humans with regards to question clarity (MD -1.21; 95% CI: -1.60,-0.82; GRADE: VERY LOW) | distractor quality (MD -1.50; 95% CI: -2.03,-0.97; GRADE: VERY LOW)
quick_read: A systematic review and network meta-analysis compared multiple-choice questions written by large language models with those written by humans for health professions education. ChatGPT 4 produced questions of similar quality to human authors on relevance, clarity and distractor quality, with mean differences close to zero.

Results varied sharply by model. Llama 2 questions scored 1.21 points lower on clarity and 1.50 points lower on distractor quality on a 10-point scale, and exploratory tests found ChatGPT 3.5 did worse than both Llama 2 and ChatGPT 4 on those measures (p < 0.001).

Every reported comparison was graded very low certainty. The authors say further studies should compare these models directly against human-written questions and give the questions to students.
limitation: All main comparisons were rated very low certainty on GRADE, and the authors say further studies are needed that directly compare these models with human-generated questions and give the questions to students.
tag: Dual reading
key_points: The review was registered with PROSPERO (CRD42025608775) and used a frequentist random-effects network meta-analysis or pairwise meta-analysis. | Ovid MEDLINE, Ovid EMBASE and Scopus were searched from inception to November 1, 2024. | Five large language models were included in the analysis. | Question quality was judged on seven outcomes, including relevance, clarity, accuracy, distractor quality, item difficulty and item discrimination. | Continuous data were converted to a 10-point scale and reported as mean differences. | ChatGPT 4 versus humans: relevance MD -0.13, clarity MD -0.03, distractor quality MD -0.10, with confidence intervals spanning zero. | Llama 2 versus humans: clarity MD -1.21 and distractor quality MD -1.50, with intervals excluding zero.
rundown: The authors searched three databases through November 1, 2024, and pooled studies in a network meta-analysis so that several models could be ranked against human question writers. Risk of bias was assessed with the MERSQI tool and certainty of evidence with GRADE network meta-analysis guidelines.

Quality was measured on seven predefined outcomes: question relevance, clarity and accuracy; distractor quality; item difficulty; and two item discrimination measures (point biserial correlation and discrimination index). Continuous results were put on a 10-point scale for comparison.

ChatGPT 4 landed within a fraction of a point of human authors on relevance, clarity and distractors, while Llama 2 trailed by more than a point on clarity and distractors. Post-hoc t-tests also placed ChatGPT 3.5 behind ChatGPT 4 and Llama 2 on those two measures. The authors conclude that ChatGPT 4 may create similar quality questions to humans, whereas ChatGPT 3.5 and Llama 2 may be worse.
sources:
- peer_reviewed | PLOS One | https://doi.org/10.1371/journal.pone.0340277 | 2026-01-02
prev: 0000000000000000000000000000000000000000000000000000000000000000

SHA-256 hash

2c4518babe81e5f39b908e1ddb275e9fa01c2e99dedc5254970b791f076ed3b9

Previous version’s hash

0000000000000000000000000000000000000000000000000000000000000000