70/100

Moderate evidence

Cite
More
EmailLinkedInX

AI-generated versus human therapist feedback in internet-delivered CBT for social anxiety disorder

Study title: AI-generated vs. human therapist feedback in internet-based CBT for social anxiety disorder: A six-arm randomized controlled trial with deception design

Evidence type
Randomized controlled trial
Sample
n = 263
Sources
1 sourcePeer-reviewed
Published
3 min read

Study title: AI-generated vs. human therapist feedback in internet-based CBT for social anxiety disorder: A six-arm randomized controlled trial with deception design

A Swedish six-arm randomized trial of 263 adults with social anxiety disorder compared written feedback in a 9-week internet CBT program drafted by an OpenAI model and reviewed by a licensed psychologist with feedback from human therapists. At post-treatment, AI feedback scored 1.15 points higher on the 68-point Social Phobia Inventory (95% CI -3.62 to 5.91; d = 0.10), a difference that was not statistically significant.

Read the study abstract

Introduction: Guided internet-delivered cognitive behavioral therapy (ICBT) is effective for social anxiety disorder (SAD), but therapist-delivered feedback limits its scalability. Whether AI-generated feedback can match human feedback, and whether patients' expectations about the source shape outcomes is unknown. Method: In a six-arm randomized controlled trial, 263 adults with SAD received psychologist-reviewed AI-generated or human feedback. They were told it came from a human or an AI which was correct or in…

56Gain side

How the two sides scoreClose call

60Problem side

The gain

In a randomized trial of 263 adults with social anxiety disorder, psychologist-reviewed AI feedback in internet CBT produced symptom reductions that did not differ significantly from human therapist feedback (1.15 points on the 0-68 SPIN scale), with no more negative effects.

The problem

In the same six-arm trial format, participants rated treatment as less credible when they imagined AI feedback (30.0 vs 35.0), and the trial could not establish non-inferiority of AI feedback because early stopping left it underpowered.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

The trial stopped early at 263 of a planned 480 participants, leaving roughly 50-60% power for the non-inferiority test. Because the confidence interval's upper bound (5.91) passed the 3.32-point margin, non-inferiority was not established. A Bayesian analysis put about 79% of the posterior inside the equivalence zone, with about 17% pointing to meaningfully worse results for AI.

Before treatment, 74% of participants rated human feedback as more credible than AI feedback, yet being told the source, or being misled about it, did not change symptoms or therapeutic alliance. The authors suggest the barrier to AI-supported internet CBT may lie in expectations rather than outcomes.

Main points

  1. Participants were randomized to four feedback arms (told AI or human, receiving AI or human feedback), unguided self-help or a waitlist; 366 were screened and 263 randomized between August 2025 and April 2026.
  2. Active treatment beat the waitlist by 13.64 SPIN points (d = -1.23); the waitlist group stayed near 40 on the scale.
  3. Feedback arms combined (n = 176) did not differ from unguided self-help (n = 44) after treatment (adjusted difference 0.96), so the benefit of feedback itself was not detected.
  4. Neither the actual source, the believed source nor their interaction affected symptoms or alliance (all p above .17).
  5. Mean negative effects were 4.4 for AI feedback versus 3.7 for human feedback (p = .29); depression and anxiety scores also did not differ by source.
  6. In an exploratory model of 86 participants, stronger alliance predicted lower later symptoms (b = -0.11, p = .037), but symptoms did not predict later alliance.
  7. Pre-treatment credibility averaged 35.0 for imagined human feedback and 30.0 for AI; 74% rated human feedback more credible and 14% favored AI.

The full story

Participants were adults scoring at least 30 on the Liebowitz Social Anxiety Scale, recruited online. The 9-week program had nine modules with weekly asynchronous written feedback. Human feedback came from 13 therapists (10 final-semester psychology students and three licensed psychologists). In the AI arms, an OpenAI model (o4, then GPT-5.1) on a dedicated Azure server in Sweden drafted each reply and a licensed psychologist reviewed it; no message was altered.

The design used deception, approved by the Swedish Ethical Review Authority. Some participants were told AI feedback came from a human, or the reverse, and all were offered full disclosure after treatment; about 20% asked. A post-treatment check showed those told human rated the source as more human than those told AI (46.5 vs 26.5), though ratings were highly variable.

Retention was limited: 96 of 176 participants in feedback arms and 20 of 44 in self-help gave post-treatment data. A completer-only analysis gave a larger AI-minus-human gap of 3.12 points (95% CI -1.24 to 7.48). The trial was registered retrospectively, on 9 April 2026.

The authors say the results cannot be extended to fully autonomous, unvetted AI feedback, and that therapist-level clustering was not modeled. Recruitment that mentioned AI may have attracted people more willing to receive it.

What this doesn’t fix

The trial stopped early at 263 of 480 planned participants, giving about 50-60% power for the primary test; AI feedback was reviewed by a licensed psychologist, post-treatment data were available for roughly half of participants, and the confidence interval cannot rule out differences larger than the margin in either direction.

Sources

1 on file
  1. Internet Interventionsdoi:10.1016/j.invent.2026.101012
    Peer-reviewed2026-10-16

Talk it over

Three questions to start on, two more if you have time
Psychologist-reviewed AI feedback came out 1.15 points higher on the 68-point scale than human feedback, while participants rated imagined AI feedback as less credible (30.0 vs 35.0). How do you weigh these two findings?

Start with a question

AI-generated versus human therapist feedback in internet-delivered CBT for social anxiety disorderhttps://truvace.com/item/ai-generated-vs-human-therapist-feedback-in-internet-based-cbt-for-social-anxiet

  1. Psychologist-reviewed AI feedback came out 1.15 points higher on the 68-point scale than human feedback, while participants rated imagined AI feedback as less credible (30.0 vs 35.0). How do you weigh these two findings?

    Answer in the thread
  2. The trial stopped at 263 of a planned 480 participants, and post-treatment data came from 96 of 176 in the feedback arms. How does that shape what you can conclude about AI feedback?

    Answer in the thread
  3. Some participants were told AI feedback came from a human, or the reverse, and offered full disclosure afterwards. If you were enrolled in this trial, how would you want the source of your feedback handled?

    Answer in the thread
If you have more time
  • Every AI reply was reviewed by a licensed psychologist and none was altered. What result would you need to see before trusting AI feedback with no human review?

    Answer in the thread
  • Recruitment that mentioned AI may have attracted people more willing to receive it. Which kinds of people with social anxiety disorder might respond differently, and how could a follow-up trial reach them?

    Answer in the thread

What readers say