Truace
Subscribe
64/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Chatbot that argued back changed more minds, but got less trust, in an 83-person study

Study title: Understanding and Supporting Online Discussion with Opinionated Chatbots

Evidence type
Controlled or validated comparison
Sample
n = 83
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2534cite or check it

Study title: Understanding and Supporting Online Discussion with Opinionated Chatbots

Researchers at the National University of Singapore and collaborators ran a controlled experiment with 83 adult students. Each person discussed whether economic growth can help environmental protection with a GPT-3.5-based chatbot for 5 to 10 rounds, then debated the same topic in text chat with a human who was actually a scripted confederate. The chatbot either opposed, reinforced or balanced the participant's views.

Read the study abstract

Opinionated chatbots are increasingly present on online platforms and have the potential to shape public discourse by influencing individuals’ viewpoints before they engage in discussions. Despite their growing presence, the impact of interacting with opinionated chatbots on subsequent online interactions remains largely unexplored. This study investigated how exposure to different types of opinionated chatbots, specifically those expressing opposing, reinforcing, or balanced viewpoints, affected participants’ s…

53Gain side

How the two sides scoreClose call

54Problem side

The gain

In a controlled experiment with 83 participants, 59% of those who prepared with a consistently opposing chatbot changed their opinion, versus 31% with a balanced chatbot and 33% with a reinforcing one.

The problem

In the same experiment, participants rated the opposing chatbot lower on trust (4.56 on a 7-point scale) than the balanced chatbot (5.49) or the reinforcing one (5.17).

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Opinion change was highest after the opposing chatbot: 59% of participants changed stance, against 31% with the balanced chatbot and 33% with the reinforcing one. Only the opposing-versus-balanced gap was statistically significant. The opposing bot also scored lowest on trust, so the design that moved opinions most was the one people liked least.

Chatbot style also carried into later talk with a human. Agreement expressions made up 49% of participants' statements after the reinforcing chatbot and 31% after the opposing one. The authors say this points to a trade-off for designers between prompting people to rethink positions and keeping trust in the tool.

Main points

  1. The 83 valid participants were split into balanced (29), opposing (27) and reinforcing (27) groups; seven of 90 responses were dropped for misunderstanding the task or inconsistent answers.
  2. Opinion shift rates were 31% (balanced), 59% (opposing) and 33% (reinforcing); only opposing versus balanced differed significantly (p<.05).
  3. Across 502 coded conversation rounds, communication patterns differed by condition (p=.013), driven by the opposing versus reinforcing contrast (Bonferroni-adjusted p=.008).
  4. Agreement made up 49% of expressions after the reinforcing chatbot versus 31% after the opposing one; the opposing chatbot did not raise disagreement.
  5. Trust scores differed by group (F(2, 80) = 10.54, p < .001); reinforcing also beat opposing on trust (p < .05).
  6. Differences in communication quality did not hold after correction for multiple tests (adjusted p = .116).
  7. Participants in the opposing group praised the chatbot's reasoning (57% of 14 commenters); some called it rigid or pushy, and two said it dropped its stance when challenged.
  8. In open-ended answers, 60% of participants said the chatbots helped with background information and critical reasoning.

The full story

Participants were students aged 21 and over recruited via Telegram and paid 12 US dollars an hour. The session lasted about an hour. Participants first chatted with one of three chatbots built on the GPT-3.5 API with a temperature of 0.2, then held a roughly 10-minute text discussion in which a confederate consistently disagreed with them and gradually moved toward consensus.

Opinion shift was measured by comparing stance before and after the chatbot chat, with two coders agreeing on a 27-response subset. Communication patterns were coded for agreement, disagreement or unclear stance across 810 rounds, of which 502 contained valid arguments. Trust and communication quality were rated on seven-point scales.

Participants who prepared with the opposing chatbot described the human partner mainly in terms of reasoning quality (68% of those with positive views), while balanced and reinforcing participants stressed friendliness. The authors suggest private, interactive disagreement with a chatbot may let people reconsider views without feeling threatened, and propose the opposing and reinforcing styles as tools for different discussion goals.

What this doesn’t fix

The experiment used a single topic (economic growth and environmental protection), a short chatbot session and a scripted human partner. Only opposing versus balanced reached significance for opinion change, and the communication-quality difference did not survive correction.

Sources

1 on file
  1. Proceedings of the ACM on Human-Computer Interactiondoi:10.1145/3817024
    Peer-reviewed2026-09-23

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Chatbot that argued back changed more minds, but got less trust, in an 83-person studyhttps://truvace.com/item/understanding-and-supporting-online-discussion-with-opinionated-chatbots

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a controlled experiment with 83 participants, 59% of those who prepared with a consistently opposing chatbot changed their opinion, versus 31% with a balanced chatbot and 33% with a reinforcing one.

    The problem In the same experiment, participants rated the opposing chatbot lower on trust (4.56 on a 7-point scale) than the balanced chatbot (5.49) or the reinforcing one (5.17).

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a controlled comparison with a sample of 83, from Proceedings of the ACM on Human-Computer Interaction.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The experiment used a single topic (economic growth and environmental protection), a short chatbot session and a scripted human partner.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Other