ChatGPT pooled meta-analysis data accurately but screened studies poorly in a validation test

Screening was the weak point. On 1,241 titles and abstracts ChatGPT reached 70.4% accuracy with 54.9% sensitivity and 80.1% specificity. In full-text selection of 475 papers, accuracy was 68.4%, sensitivity 75.6% and specificity 66.8%. Data pooling went much better: all five forest plots were produced, pooled mean differences matched in four of five analyses, and tau-squared values differed by 0.01 to 0.05 in three.

The authors conclude that the tool could speed up the number-crunching part of systematic reviews, while selection of studies still needs human oversight. They note that the original meta-analysis may have been in the model's training data, so performance on new datasets is untested.

The study was a prospectively conducted validation exercise exempt from institutional review board review. Prompts were written to mirror the eligibility criteria of Klasova et al: adults over 18 with chronic pain, spinal cord stimulator implantation, and emotional functioning outcomes measured by validated questionnaires. The authors pasted the full list of titles and abstracts, and then each full manuscript, into ChatGPT-4o.

Before screening, 2,186 records were identified and duplicates were removed, including 43 extra duplicates the authors removed to help the AI, leaving 1,241 abstracts instead of the original 1,284. Of 495 papers sought for full-text review, two had no full text and 16 duplicates were removed, leaving 475.

For pooling, ChatGPT was given each study's mean difference, 95% confidence interval and weight and asked for a random-effects pooled estimate, I² and tau-squared. Pooled mean differences and confidence intervals matched the published analysis, heterogeneity estimates matched in all five, and tau-squared matched in two.

The authors list several constraints: only ChatGPT's first answer was recorded although repeated screening sometimes gave different answers, risk-of-bias and certainty-of-evidence steps were not tested, and only one pain-medicine review was used. They also say future work should test unpublished datasets alongside human reviews in progress.