Chatbot practice boosted language learners' speaking scores, a review of 16 experiments finds

The studies came from countries including Taiwan, China, Jordan, Egypt, Iran, Korea and Turkiye, and covered primary pupils, high schoolers and university students. Sample sizes ranged from 42 to 93 per study. Results varied widely: 14 of 17 effect sizes were positive and 3 were negative, and heterogeneity was high (I² = 91.89).

One study, El-Said (2023), reported an outlying effect of 5.825. Without it, the pooled estimate fell to 0.793. Neither education level nor intervention duration differed significantly, though duration came close (p = 0.053), with long-term programmes averaging 1.864.

The authors followed a PRISMA process, with two researchers searching databases between 15 and 30 January 2025 and coding the studies independently, reaching 94% agreement. Effects were computed as Hedges' g in Comprehensive Meta-Analysis software using a random-effects model.

The included trials used tools such as Duolingo, Replika, LINE, Mondly and TPBOT, with control groups taught by other methods. Sample sizes ran from 42 to 93 learners. Six effect sizes exceeded 1, five fell between 0.5 and 1 and six were below 0.5.

Publication bias checks gave mixed signals: Egger's test found no significant bias (p = .225), but the funnel plot was asymmetric, and a trim-and-fill adjustment suggested four missing studies and a 26% difference between observed and corrected values. Fail-safe N values were 951 and 101.

The authors say the field is still early. Chatbot type could not be tested as a moderator because too few studies existed, and speaking sub-skills such as fluency, accuracy and pronunciation were combined into one score.