aceModerate evidence
More
Chatbots give students a small to moderate learning boost, a review of 62 studies finds
Study title: Chatbots in education: Hype or help? A meta-analysis
- Evidence type
- Systematic review or meta-analysis
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2841cite or check it
Study title: Chatbots in education: Hype or help? A meta-analysis
A meta-analysis following PRISMA guidelines pooled 135 effect sizes from 62 studies across varied educational settings to test whether chatbots improve learning performance. Before any correction, it found a large positive effect. After controlling for publication bias, the effect shrank to small to moderate.
Read the study abstract
Artificial intelligence integration in education, primarily through chatbots , has emerged as a potential solution to address the challenges of catering to students' diverse learning backgrounds. This meta-analysis examined chatbot effectiveness in education, driven by amplified interest since ChatGPT's introduction in 2022. Adhering to PRISMA guidelines, we analyzed 135 effect sizes from 62 studies across diverse educational contexts. Initial results revealed a large positive effect of chatbots on learning perf…
The gain
A meta-analysis of 135 effect sizes from 62 studies found chatbots had a small to moderate positive effect on student learning performance after publication-bias correction, with the largest effects in STEM, text-based and longer interventions.
In brief
The shrinkage matters because interest in chatbots has surged since ChatGPT's introduction in 2022, and the authors say the impact may be overestimated in the published literature. They conclude that chatbots can support performance but need targeted implementation.
Effects varied by context. Interaction mode, field of study, intervention duration and educational level were significant moderators, while chatbot type and cultural differences were not.
Main points
- The review analysed 135 effect sizes drawn from 62 studies in diverse educational contexts.
- The uncorrected result was a large positive effect on learning performance.
- After controlling for publication bias, the effect fell to a significant small to moderate one.
- Text-based interactions worked better than other interaction modes.
- STEM disciplines showed stronger chatbot effects than other fields.
- Longer interventions and lower educational levels were linked to the strongest effects.
- Neither chatbot type nor cultural differences significantly influenced learning performance.
The full story
The authors followed PRISMA guidelines to pool 135 effect sizes from 62 studies, motivated by the jump in interest in chatbots in education since ChatGPT arrived in 2022. The aim was to test whether chatbots help address students' diverse learning backgrounds.
The headline result changed once publication bias was addressed: an initially large positive effect on learning performance became a significant small to moderate effect. The authors say the impact may be overestimated because of this bias.
Four factors significantly moderated the effect: interaction mode, field of study, intervention duration and educational level. The strongest effects came with text-based interactions, STEM subjects, longer interventions and lower educational levels, while chatbot type and cultural differences made no significant difference.
Sources
1 on file- Learning and Individual Differencesdoi:10.1016/j.lindif.2025.102646
Talk it over
Who gets this benefit, and who was left out of the study or story?
Chatbots give students a small to moderate learning boost, a review of 62 studies findshttps://truvace.com/item/chatbots-in-education-hype-or-help-a-meta-analysis
Who gets this benefit, and who was left out of the study or story?
The gain A meta-analysis of 135 effect sizes from 62 studies found chatbots had a small to moderate positive effect on student learning performance after publication-bias correction, with the largest effects in STEM, text-based and longer interventions.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a systematic review or meta-analysis, from Learning and Individual Differences.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat The authors say chatbots' impact may be overestimated because of publication bias, so the pooled effect on learning performance is likely smaller than early results suggested.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say