aceModerate evidence
Published recordScienceThe Trace
More
Human versus artificial intelligence: evaluating ChatGPT’s performance in conducting published systematic reviews with meta-analysis in chronic pain research
- Truvace record
- TRV-2026-2791raw data (JSON)
- Version
- v1Published
- First published
- Evidence type
- Benchmark or lab evaluation
- Sources
- 1 sourcePeer-reviewed
- sha256
- 7c36395a0ca7…check it
Introduction Artificial intelligence (AI), particularly large-language models like Chat Generative Pre-Trained Transformer (ChatGPT), has demonstrated potential in streamlining research methodologies. Systematic reviews and meta-analyses, often considered the pinnacle of evidence-based medicine, are inherently time-intensive and demand meticulous planning, rigorous data extraction, thorough analysis, and careful synthesis. Despite promising applications of AI, its utility in conducting systematic reviews with me…
54Gain side
How the two sides scoreClose call
53Problem side
The gain
In a validation study replicating a published spinal cord stimulation meta-analysis, ChatGPT-4o pooled mean differences with 100% accuracy for four of five analyses and matched all 95% confidence intervals and I² heterogeneity values.
The problem
In the same spinal cord stimulation meta-analysis replication, ChatGPT-4o screened 1,241 titles and abstracts with 54.9% sensitivity and 70.4% accuracy, producing 214 false negatives, and reached only 68.4% accuracy in full-text selection.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
What this doesn’t fix
The test used one previously published pain-medicine meta-analysis that may have been in ChatGPT's training data, only the first response was recorded, and risk-of-bias assessment was not tested, so performance on new reviews is unknown.
Sources
1 on file- Regional Anesthesia & Pain Medicinedoi:10.1136/rapm-2024-106358
Cite this record
v1Truvace Impact Record TRV-2026-2791, v1: “Human versus artificial intelligence: evaluating ChatGPT’s performance in conducting published systematic reviews with meta-analysis in chronic pain research.” Truvace, 2026-11-12. /record/TRV-2026-2791 (accessed on the day you copy it). sha256 7c36395a0ca7abf3…
Change history
1 versionEvery change made to this record since it was first published, and why. Nothing has changed since it went up.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2791 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.