aceModerate evidence
More
Chatbots modestly boost learning, a review of 28 studies finds
Study title: The effect of chatbots on learning: a meta-analysis of empirical research
- Evidence type
- Systematic review or meta-analysis
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2804cite or check it
Study title: The effect of chatbots on learning: a meta-analysis of empirical research
A meta-analysis in the Journal of Research on Technology in Education pooled 28 eligible reports, giving 31 individual effect sizes, to estimate how chatbots affect learning. It found a significant, medium overall effect of g = .48.
Read the study abstract
This meta-analysis aimed to comprehensively review empirical studies on the effect of chatbots on learning and quantitatively synthesize their findings to produce an overall effect size. Searching several databases yielded 28 eligible reports with 31 individual effect sizes. The results revealed a significant and medium effect (g = .48) of chatbots on learning. Further analyses indicated four significant moderators: the type of instruction in the comparison group, experimental duration, chatbot type, and chatbot…
The gain
A meta-analysis of 28 empirical reports (31 effect sizes) found chatbots had a medium positive effect on learning (g = .48), with the largest effects in single-session use and for chatbots handling frequently asked questions.
In brief
The effect was not uniform. Four factors changed it: the type of instruction in the comparison group, how long the experiment lasted, the type of chatbot and the tasks the chatbot performed.
The largest effects appeared when the comparison group had no specific support, when the experiment lasted just one session, and when chatbots were task-focused and answered frequently asked questions. The authors conclude chatbots can be more effective in certain cases.
Main points
- The analysis covered 28 eligible reports that together supplied 31 effect sizes.
- The overall effect of chatbots on learning was significant and medium (g = .48).
- Four moderators were significant: comparison-group instruction, experiment duration, chatbot type and chatbot task.
- Effects were highest when the comparison group got no specific support.
- Single-session experiments produced the highest effect sizes.
- Task-focused chatbots that handled frequently asked questions did best.
The full story
The researchers searched several databases for empirical studies on chatbots and learning and pooled their results quantitatively. They identified 28 eligible reports with 31 individual effect sizes, producing an overall effect of g = .48.
Moderator analyses showed which conditions shaped the result. Four were significant: what instruction the comparison group received, how long the experiment ran, the kind of chatbot and the tasks it performed.
The strongest results came when comparison students had no specific support, when the experiment lasted only one session, and when chatbots were task-focused and handled frequently asked questions. The authors say these findings suggest chatbots work better in some cases than others within their contribution area to learning.
Sources
1 on file- Journal of Research on Technology in Educationdoi:10.1080/15391523.2023.2255698
Talk it over
Who gets this benefit, and who was left out of the study or story?
Chatbots modestly boost learning, a review of 28 studies findshttps://truvace.com/item/the-effect-of-chatbots-on-learning-a-meta-analysis-of-empirical-research
Who gets this benefit, and who was left out of the study or story?
The gain A meta-analysis of 28 empirical reports (31 effect sizes) found chatbots had a medium positive effect on learning (g = .48), with the largest effects in single-session use and for chatbots handling frequently asked questions.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a systematic review or meta-analysis, from Journal of Research on Technology in Education.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat The effect varied significantly by comparison condition, duration, chatbot type and task, and the highest effects came from one-session experiments, so the overall figure does not apply equally across settings.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say