aceEmerging evidence
More
AI chatbots matched human coders on 92% or more of psychology study codes, a test of 60 papers finds
Study title: AI-accelerated meta-analysis in psychology: Large language models code study properties with high accuracy
- Evidence type
- Benchmark or lab evaluation
- Sample
- n = 60
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2755cite or check it
Study title: AI-accelerated meta-analysis in psychology: Large language models code study properties with high accuracy
A psychologist at the University of Münster tested whether two AI models could act as a second coder for meta-analyses, the step in which researchers pull study characteristics out of papers. GPT-5 matched the original authors' codes 93% of the time and Gemini 2.5 Pro 92% of the time, on moderators that could be coded from the articles themselves.
Read the study abstract
Theoretically driven meta-analyses often involve testing whether study characteristics moderate an effect in a pattern that supports one of multiple competing theoretical accounts. To conduct such analyses, researchers must manually extract, i.e., code, these characteristics from sometimes hundreds of studies. Ideally, studies should be coded by two researchers independently to prevent errors and biases from compromising the dataset's integrity. The laborious nature of this task, however, means that meta-analyst…
The gain
In a test on 60 articles drawn from three published psychology meta-analyses, GPT-5 matched the original authors' study codes 93% of the time and Gemini 2.5 Pro 92% of the time, with Cohen's kappa of .75 to .96.
In brief
The test covered three Psychological Bulletin meta-analyses, on memory judgments, attitudes toward older men and women, and social class and prosociality. For each, 20 randomly chosen studies from different sources were coded. The models worked from a coding manual and the article text, and returned structured data rather than chat replies.
Meta-analysts often double-code only a share of their studies; the three original teams double-coded 20%, 27% and all of theirs. The author argues AI double-coding of every study could make datasets more reliable. Results come from only three meta-analyses and two models, and the author calls for larger studies.
Main points
- The author selected three meta-analyses from Psychological Bulletin Volume 161 after screening 18 articles; 20 randomly chosen studies from each, one per source, made up the coding dataset.
- GPT-5 averaged 93% agreement with the authors' codes on Ingendahl et al.; excluding two continuous moderators raised this to 98%, versus 95% for Gemini 2.5 Pro.
- On Shakeri and North, Gemini 2.5 Pro reached 89% agreement and GPT-5 90%; 62% of the 29 disagreements with the authors' codes were shared by both models.
- Both models scored 93% on Wu et al., and agreement between the two models was 94%.
- Cohen's kappa was almost perfect for Ingendahl et al. (.90 for Gemini, .96 for GPT-5) and Wu et al. (.86 and .84), and substantial for Shakeri and North (.75 and .78).
- A second GPT-5 run about three months later agreed with the first 96% of the time (minimum 85%).
- An earlier study found older models coded study properties with 69% accuracy overall, versus the 92% or higher found here.
The full story
Study coding in psychology often requires judgment calls, such as whether a measure captured prosocial behavior or only intention. The author notes that a simulation study found miscategorizing just 2% of moderator codes can substantially affect discovery rates.
Part of the disagreement came from missing information. For Ingendahl et al., some codes had come from raw data or from emailing the source authors, so the models could not recover them. For Shakeri and North, an initial round exposed ambiguities in the coding manual, so the author amended it and generated new codes from a second random sample.
The author reports that some disagreements reflected ambiguity rather than model error, for example the models coded an election as a non-work context where the original authors coded it as work. Agreement was lowest for Shakeri and North, who double-coded the fewest studies.
The study used GPT-5 and Gemini 2.5 Pro through programming packages rather than chat windows, with Gemini's temperature set to zero. The author says this avoids manual transfer errors, and shared the data and analysis scripts on the Open Science Framework.
Sources
1 on file- Behavior Research Methodsdoi:10.3758/s13428-026-03020-1
Talk it over
Who gets this benefit, and who was left out of the study or story?
AI chatbots matched human coders on 92% or more of psychology study codes, a test of 60 papers findshttps://truvace.com/item/ai-accelerated-meta-analysis-in-psychology-large-language-models-code-study-prop
Who gets this benefit, and who was left out of the study or story?
The gain In a test on 60 articles drawn from three published psychology meta-analyses, GPT-5 matched the original authors' study codes 93% of the time and Gemini 2.5 Pro 92% of the time, with Cohen's kappa of .75 to .96.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a benchmark or lab test with a sample of 60, from Behavior Research Methods.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat The test covered only three meta-analyses and two models, and agreement was measured against authors' codes that were often not double-coded and so may contain errors; the author says larger-scale studies are needed.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say