Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

A review of 7 studies finds AI messages persuade about as well as people

Study title: A meta-analysis of the persuasive power of large language models

Evidence type
Systematic review or meta-analysis
Sample
n = 17,422
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2706cite or check it

Study title: A meta-analysis of the persuasive power of large language models

Researchers pooled 12 effect sizes from 7 experiments, covering 17,422 participants, that directly compared persuasion by large language models with persuasion by human communicators. On average they found no significant difference between the two.

Read the study abstract

Large language models (LLMs) are increasingly used for persuasion, such as in political communication and marketing, where they affect how people think, choose, and act. Yet, empirical findings on the effectiveness of LLMs in persuasion compared to humans remain inconsistent. The aim of this study was to systematically review and meta-analytically assess whether LLMs differ from humans in persuasive effectiveness, and under which contextual conditions LLMs are particularly effective. We identified 7 studies with…

A meta-analysis of the persuasive power of large language models
"Biologist using inverted microscope 01" by Mateus S. Figueiredo, CC BY 4.0.

The gain

A meta-analysis of 7 studies with 17,422 participants found no significant overall difference in persuasiveness between LLM-generated and human-written messages, so the models matched human persuaders on average.

In brief

The pooled result hides wide variation between studies. The authors say how persuasive a model is seems to depend on the model, on whether the exchange is a single message or an interactive conversation, and on the topic, though no single factor was significant on its own.

The evidence base is small and narrow. Participants came mainly from English-speaking countries and online panels, and the authors treat the combined-factor findings as possibly overfitted.

Main points

  1. The review searched Web of Science, ACM Digital Library, arXiv, SSRN and OSF for English-language studies published after December 2022, up to May 22, 2025.
  2. Seven studies with 12 independent effect sizes met the strict criteria; four papers were dropped because their authors did not supply missing statistics.
  3. The pooled difference in persuasiveness between LLMs and humans was very small and not statistically significant, and the result held when only peer-reviewed studies were analysed.
  4. Heterogeneity between studies was substantial, which the authors read as persuasiveness depending on context.
  5. Across the 12 estimates, 7 used one-shot messages and 5 used interactive dialogue; models ranged from GPT-3 and GPT-3.5 to GPT-4 and Claude Sonnet.
  6. In a combined model, interactive setups beat one-shot messages, Claude 3.x trailed GPT-4.x, and political topics were less persuasive than health topics.
  7. Early studies from 2021 and 2022 leaned toward LLMs being somewhat less persuasive than humans, but adding 2024 and 2025 studies moved the cumulative estimate toward zero and slightly positive.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Persuasive effectiveness of large language model messages compared with human-written messages

    Hedges' g 0.02 (95% CI −0.048 to 0.093)

    7 studies · 17,422 people

    The review’s own words
    The overall effect size was very small and nonsignificant (g = 0.02, p = .530, 95% CI [−0.048, 0.093]).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors followed PRISMA 2020 guidelines and required a direct between-subjects comparison of LLM-generated and human-generated content. None of the included studies reported standardized effect sizes, so the authors computed Hedges' g from means, standard deviations and test statistics. They used a random-effects model with a restricted maximum-likelihood estimator.

In single-factor analyses, no moderator reached significance: not the model (GPT-3.x with 5 effect sizes, GPT-4.x with 4, Claude 3.x with 3), not the format, and not the domain. Politics and health were almost equally effective, while a catch-all category of propaganda, quiz and mixed tasks looked slightly weaker. The authors say small subgroups may have left these tests underpowered.

When all factors were entered together, the model explained a large share of between-study variance and residual heterogeneity became low. GPT-3.x then outperformed GPT-4.x once domain and interaction type were controlled for, which suggests the apparent GPT-4 advantage in the single-factor analysis reflected study design. The authors warn the drop in heterogeneity may reflect overfitting.

Across the reviewed studies, human messages tended to be more emotionally vivid and personally engaging, while LLM messages leaned on analytical reasoning and coherence. Only one included study tested personalization, and it showed a notably high positive effect. Leave-one-out checks left the non-significant pooled result unchanged, and trim-and-fill added no missing studies.

Sources

1 on file
  1. Scientific Reportsdoi:10.1038/s41598-025-30783-y
    Peer-reviewed2025-12-12

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

A review of 7 studies finds AI messages persuade about as well as peoplehttps://truvace.com/item/a-meta-analysis-of-the-persuasive-power-of-large-language-models

  1. Who gets this benefit, and who was left out of the study or story?

    The gain A meta-analysis of 7 studies with 17,422 participants found no significant overall difference in persuasiveness between LLM-generated and human-written messages, so the models matched human persuaders on average.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 17,422, from Scientific Reports.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Only 7 studies with 12 effect sizes were eligible, which limits the subgroup analyses.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Science