Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Programming students did better with AI chatbots, a review of 32 studies finds

Study title: Do AI Chatbots Improve Students’ Learning Performance in Programming Education? Evidence from a Meta-Analysis

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2814cite or check it

Study title: Do AI Chatbots Improve Students’ Learning Performance in Programming Education? Evidence from a Meta-Analysis

A meta-analysis of 32 empirical studies published between 2015 and 2025 found that AI chatbots improved students' programming learning. The pooled effect was g+ = 0.538 on posttest performance, which the authors call small-to-medium, and g+ = 0.650 on practice performance, medium-to-large.

Read the study abstract

AI chatbots have emerged as innovative educational tools and drawn increasing attention from educators and researchers in programming education. Although previous research has highlighted potentials of applying AI chatbots in programming education, there is a lack of empirical evidence to understand the overall effects of using AI chatbots in programming learning as well as the critical factors that influence the effects. To fill this gap, this study conducted a meta-analysis of 32 empirical studies published be…

The gain

A meta-analysis of 32 studies from 2015 to 2025 found students who learned programming with AI chatbots scored higher, with an effect of g+ = 0.538 on posttests and g+ = 0.650 on practice tasks.

In brief

The authors say earlier work pointed to the potential of chatbots in programming classes but lacked empirical evidence on their overall effect. The review also looked at which factors change the size of the benefit.

Two moderators mattered for posttest results: how the original study was designed and how many students shared each chatbot. True experiments showed larger effects than quasi-experiments, and one chatbot per student worked substantially better than one chatbot for many students.

Main points

  1. The meta-analysis pooled 32 empirical studies published between 2015 and 2025.
  2. Posttest performance rose by g+ = 0.538 (95% CI .202 to .873, p < .01).
  3. Practice performance rose by g+ = 0.650 (95% CI .330 to .970, p < .001).
  4. Estimates came from robust variance estimation models.
  5. True experimental designs produced significantly larger effects than quasi-experimental designs.
  6. A 1:1 chatbot-to-student ratio was substantially more effective than a 1:N ratio.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Programming practice performance with AI chatbots compared with control conditions

    Hedges' g 0.650 (95% CI .330 to .970)

    32 studies

    The review’s own words
    a medium-to-large effect on practice performance ( g+ = 0.650, 95% CI [.330, .970], p < .001)

  • Programming posttest performance with AI chatbots compared with control conditions

    Hedges' g 0.538 (95% CI .202 to .873)

    32 studies

    The review’s own words
    The results indicated a small-to-medium effect on posttest performance ( g+ = 0.538, 95% CI [.202, .873], p < .01)

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The review pooled 32 empirical studies from 2015 to 2025 to estimate how much AI chatbots change programming learning performance. Using robust variance estimation, it reported g+ = 0.538 on posttests and g+ = 0.650 on practice performance.

Moderator analyses found that research design and chatbot-to-student ratio significantly influenced posttest performance. Studies with true experimental designs showed larger effects than quasi-experimental ones.

The authors found a 1:1 chatbot-student ratio substantially more effective than a 1:N ratio, and say the findings offer practical guidance for building chatbots into instructional design.

Sources

1 on file
  1. Journal of Educational Computing Researchdoi:10.1177/07356331261424211
    Peer-reviewed2026-02-17

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

Programming students did better with AI chatbots, a review of 32 studies findshttps://truvace.com/item/do-ai-chatbots-improve-students-learning-performance-in-programming-education-ev

  1. Who gets this benefit, and who was left out of the study or story?

    The gain A meta-analysis of 32 studies from 2015 to 2025 found students who learned programming with AI chatbots scored higher, with an effect of g+ = 0.538 on posttests and g+ = 0.650 on practice tasks.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from Journal of Educational Computing Research.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Effects were significantly larger in true experimental designs than in quasi-experimental designs, so the pooled estimates vary with how the underlying studies were run.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education