Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 106 experiments finds human-AI teams often trail the best solo performer

Study title: When combinations of humans and AI are useful: A systematic review and meta-analysis

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2796cite or check it

Study title: When combinations of humans and AI are useful: A systematic review and meta-analysis

A preregistered systematic review and meta-analysis in Nature Human Behaviour pooled 106 experiments (370 effect sizes) published between January 2020 and June 2023. On average, human-AI combinations performed worse than the better of humans alone or AI alone (Hedges' g = -0.23), a small negative effect.

Read the study abstract

Inspired by the increasing use of artificial intelligence (AI) to augment humans, researchers have studied human-AI systems involving different tasks, systems and populations. Despite such a large body of work, we lack a broad conceptual understanding of when combinations of humans and AI are better than either alone. Here we addressed this question by conducting a preregistered systematic review and meta-analysis of 106 experimental studies reporting 370 effect sizes. We searched an interdisciplinary set of dat…

56Gain side

How the two sides scoreClose call

58Problem side

The gain

A meta-analysis of 106 experiments (370 effect sizes) found human-AI teams outperformed humans working alone (Hedges' g = 0.64), and content-creation tasks showed gains over the best solo performer (g = 0.19, not statistically significant).

The problem

A meta-analysis of 106 experiments (370 effect sizes) found human-AI combinations did worse than the better of humans or AI alone (g = -0.23), with decision tasks losing ground (g = -0.27) and AI-outperforming-human tasks worse still (g = -0.54).

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

The same pooled data show human-AI teams did beat humans working alone (g = 0.64, medium to large). The gap between the two results means the AI helped people, but often the AI by itself would have done as well or better.

Task type and relative skill mattered. Decision tasks, about 85% of effect sizes, showed losses (g = -0.27), while creation tasks, about 10%, showed a positive but non-significant gain (g = 0.19). When humans outperformed AI alone, teams gained (g = 0.46); when AI outperformed humans, teams lost (g = -0.54).

Main points

  1. The search found 5,126 papers; 74 met the inclusion criteria, covering 106 experiments and 370 effect sizes from 2020 to mid-2023.
  2. Each study had to report performance for humans alone, AI alone and the human-AI team, and the three-level model accounted for dependent effect sizes.
  3. In a fake hotel review task cited from Cabrera and colleagues, AI alone scored 73% accuracy, humans 55% and the team 69%; for bird images, AI scored 73%, humans 81% and the team 90%.
  4. Explanations and AI confidence displays did not significantly change team performance, nor did participant type or division of labour.
  5. Only 3 of the 100+ experiments assigned subtasks to humans and AI in advance; their pooled synergy (g = 0.22) was not statistically significant.
  6. Heterogeneity was very high (I2 = 97.7% for synergy), and results held in sensitivity checks, including excluding 11 outliers (g = -0.25).
  7. Egger's test suggested publication bias favouring studies where teams beat humans alone, but not for the synergy outcome.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Task performance of human-AI combinations compared with the best of humans or AI alone

    Hedges' g -0.23 (95% CI -0.39 to -0.07)

    106 studies · search to 30 June 2023

    The review’s own words
    First, we found that, on average, human-AI combinations performed significantly worse than the best of humans or AI alone (Hedges' g = -0.23; 95% confidence interval, -0.39 to -0.07).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors defined two outcomes: synergy, where the team beats both humans alone and AI alone, and augmentation, where the team beats humans alone. Effect sizes were computed as Hedges' g across accuracy, error and quality measures, drawing on databases from computing, information systems and Web of Science.

Task type split the results: decision tasks, where participants chose among set options, lost performance when combined (g = -0.27), while open-ended creation tasks leaned positive (g = 0.19, 95% CI -0.09 to 0.48) with only 34 effect sizes. The authors suggest generative AI creation tasks are an under-studied area.

Relative skill was a strong factor. When the human beat the AI alone, the team outperformed both (g = 0.46); when AI beat the human, teams lost to the AI (g = -0.54), though they still improved on humans alone (g = 0.74). Over 95% of systems had humans making the final decision.

The authors recommend designing processes that give AI only the subtasks it does better, and reporting AI-alone performance, cost, time and error severity. They say the findings do not mean combining humans and AI is a bad idea.

What this doesn’t fix

The results apply only to laboratory-style experiments that reported humans alone, AI alone and the team, which may not reflect real-world deployments; heterogeneity was very high, and tests suggested possible publication bias for the augmentation outcome.

Sources

1 on file
  1. Nature Human Behaviourdoi:10.1038/s41562-024-02024-1
    Peer-reviewed2024-10-28

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Review of 106 experiments finds human-AI teams often trail the best solo performerhttps://truvace.com/item/when-combinations-of-humans-and-ai-are-useful-a-systematic-review-and-meta-analy

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain A meta-analysis of 106 experiments (370 effect sizes) found human-AI teams outperformed humans working alone (Hedges' g = 0.64), and content-creation tasks showed gains over the best solo performer (g = 0.19, not statistically significant).

    The problem A meta-analysis of 106 experiments (370 effect sizes) found human-AI combinations did worse than the better of humans or AI alone (g = -0.23), with decision tasks losing ground (g = -0.27) and AI-outperforming-human tasks worse still (g = -0.54).

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from Nature Human Behaviour.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The results apply only to laboratory-style experiments that reported humans alone, AI alone and the team, which may not reflect real-world deployments; heterogeneity was very high, and tests suggested possible publication bias for the augmentation outcome.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Science