Meta-analysis of 8,282 learners shows AI-guided language lessons beat usual teaching modestly

Across 26 within-group samples (N = 2,262), learners improved substantially over time (d = 1.18). Across 35 between-group samples (N = 6,020), AI-guided learning outperformed business-as-usual conditions by a smaller margin (d = 0.39).

Moderator analyses found that systems built with machine learning and hybrid approaches had larger effects than rule-based systems. The authors note that rule-based systems may be easier to interpret pedagogically, since it is clearer how their predictions are made.

The meta-analysis synthesized recent empirical work on AI that assesses learning progress and recommends individual instruction for language learners, drawing on 17 research projects.

Two kinds of comparison were pooled. Within-group designs, which track learners' progress over time, gave an effect of d = 1.18 across 26 samples. Between-group designs, which compare AI-guided learners with business-as-usual conditions, gave a smaller effect of d = 0.39 across 35 samples.

Moderator analyses showed machine learning and hybrid systems were more impactful than rule-based ones, though the authors say rule-based systems may be more useful for understanding how predictions are made pedagogically.