Meta-analysis of 27 studies finds AI tutoring agents modestly raise student results

The size of the effect varied by setting. It was larger in collectivist cultures and in teacher-directed learning than in self-directed learning, and dialogue style mattered: multimodal dialogue showed the highest pedagogical potential.

Grade level, gender, learning domain, learning duration and the instructional role of the agent did not change the effect. The authors discuss design and application recommendations, and say the question of benefit had been debated.

The researchers set out to settle a debate over whether generative AI-powered pedagogical agents help learners' academic performance. They pooled 27 experimental and quasi-experimental studies spanning 2015 to 2025 and found a significant overall effect of g = 0.401.

Moderator analyses showed where the agents worked best. Effects were greater in collectivist cultures, and teacher-directed learning outperformed self-directed learning. The dialogue modality also mattered, with multimodal dialogue showing the highest potential.

No significant moderating effect was found for grade level, gender, learning domain, learning duration or whether the agent played an instructional role. The paper closes with recommendations for designing and applying these agents.