Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Meta-analysis of 8 studies finds chatbots lift students' science scores, but results vary widely

Study title: Meta-Analysis on the Effect Size of Chatbot Integration in Student Science Performance

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2822cite or check it

Study title: Meta-Analysis on the Effect Size of Chatbot Integration in Student Science Performance

A meta-analysis from National University in Manila pooled eight quasi-experimental studies on chatbot use in science classes, drawn from an initial 1,056 records. It found a pooled Hedges' g of 2.41 (95% CI 1.05–3.76, p < 0.001) favouring chatbot-assisted instruction over traditional teaching.

Read the study abstract

This meta-analysis examines the effectiveness of chatbot integration in science education by synthesizing quantitative findings from selected peer-reviewed studies. Employing a rigorous meta-analytic approach, eight studies published between 2007 and 2024 were analyzed using a random-effects model. The inclusion criteria focused on quasi-experimental research measuring student performance in science subjects. Effect sizes were calculated using Hedges’ g, with additional analyses conducted to assess heterogeneity…

The gain

A meta-analysis of eight quasi-experimental studies found chatbot-assisted science instruction beat traditional teaching (pooled Hedges' g = 2.41, 95% CI 1.05–3.76), with AI-driven chatbots at g = 2.97 versus 0.82 for rule-based ones.

In brief

The studies covered Grade 6 to university students in Taiwan, South Korea, Pakistan, the USA and Ghana, with sample sizes from 32 to 192. AI-driven chatbots (six studies) averaged g = 2.97, while rule-based chatbots (two studies) averaged 0.82, a range that included zero. The group difference was statistically significant (p = 0.020).

The evidence is uneven. Individual effects ranged from 0.35 to 5.94, heterogeneity was I² = 97.67%, and Egger's test pointed to small-study effects consistent with publication bias. The authors say the findings call for cautious interpretation and structured teacher guidance.

Main points

  1. A search of 1,056 records yielded nine eligible studies; one with an extreme effect (g = 19.283) was excluded, leaving eight in the final analysis.
  2. The pooled effect was g = 2.41 (95% CI 1.05–3.76; z = 3.48, p < 0.001) in favour of chatbot-assisted groups.
  3. Six AI-driven chatbot studies averaged g = 2.97, versus 0.82 (CI −0.048 to 1.678) for two rule-based chatbot studies.
  4. The largest single effects were g = 5.94 (Chang, Kuo and Hwang, 32 students) and g = 4.53 (Essel et al., 68 students in Ghana); Lin and Ye (g = 0.35) and Prondoza and Panoy (g = 0.36) had no significant effect.
  5. Leave-one-out checks kept the pooled effect between 1.96 and 2.71, and every run stayed significant.
  6. Heterogeneity was very high (I² = 97.67%, Q(7) = 139.95), and sample size did not explain it in a meta-regression (p = 0.436).
  7. Egger's test was significant (p = 0.0003), though trim-and-fill imputed no missing studies.
  8. The authors report that university students and conceptual subjects such as biology and chemistry showed larger gains, though chatbots may be weaker for higher-order thinking.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Student science performance with rule-based chatbots compared with control

    Hedges' g 0.815 (95% CI −0.048 to 1.678)

    search to first quarter of 2024

    The review’s own words
    The results indicated that AI-driven chatbots demonstrated a significantly larger effect size (Hedges’ g = 2.969, 95% CI: 1.375, 4.563) compared to rule-based chatbots (Hedges’ g = 0.815, 95% CI: −0.048, 1.678).

  • Student science performance with chatbot-assisted instruction compared with traditional teaching

    Hedges' g 2.41 (95% CI 1.05 to 3.76)

    search to first quarter of 2024

    The review’s own words
    The pooled effect size, Hedges’s g, was 2.41 (95% CI: 1.05–3.76), indicating a statistically significant and large effect of the intervention (Table 3).

  • Student science performance with AI-driven chatbots compared with control

    Hedges' g 2.969 (95% CI 1.375 to 4.563)

    search to first quarter of 2024

    The review’s own words
    The results indicated that AI-driven chatbots demonstrated a significantly larger effect size (Hedges’ g = 2.969, 95% CI: 1.375, 4.563) compared to rule-based chatbots (Hedges’ g = 0.815, 95% CI: −0.048, 1.678).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors searched Google Scholar, ERIC, JSTOR and Publish or Perish for studies from 2007 to early 2024. They required quasi-experimental designs, science performance as the outcome and enough data to compute effect sizes. Of 1,056 records, nine qualified, and one was dropped as an outlier because it pushed heterogeneity to 99.66%.

The included studies were small: Chang et al. had 36 university students, Lin and Ye 34 Grade 7 students, Lee et al. 192 Grade 6 students and Essel et al. 68 university students in Ghana. Weights ranged only from 11.06% to 13.01%, so the pooled estimate leans on a handful of small trials.

The authors attribute AI-driven chatbots' edge to real-time feedback, adaptive responses and self-regulated learning support. They say university students and conceptual subjects such as biology and chemistry gained more, and studies using standardized exams showed larger effects than those using self-reported gains. These explanations are interpretive and were not separately tested in the reported subgroup analysis.

The authors recommend future work on long-term retention and critical thinking, standardized outcome measures and replication, and say chatbots work best inside a structured teaching framework with teacher guidance.

Sources

1 on file
  1. International Journal of Information and Education Technologydoi:10.18178/ijiet.2026.16.6.2633
    Peer-reviewed2026-01-01

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

Meta-analysis of 8 studies finds chatbots lift students' science scores, but results vary widelyhttps://truvace.com/item/meta-analysis-on-the-effect-size-of-chatbot-integration-in-student-science-perfo

  1. Who gets this benefit, and who was left out of the study or story?

    The gain A meta-analysis of eight quasi-experimental studies found chatbot-assisted science instruction beat traditional teaching (pooled Hedges' g = 2.41, 95% CI 1.05–3.76), with AI-driven chatbots at g = 2.97 versus 0.82 for rule-based ones.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from International Journal of Information and Education Technology.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Heterogeneity was very high (I² = 97.67%) and Egger's test indicated small-study effects consistent with possible publication bias, so the pooled effect should be read cautiously.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education