AI's STEM learning boost mostly vanishes after bias correction, a review of 49 studies finds

After correcting for publication bias with a Robust Bayesian Meta-Analysis, the average effect shrank to 0.076 ± 0.254, so the headline benefit is largely a product of which studies get published. Funnel plot asymmetry also pointed to possible publication bias.

The size of the effect depended on what was measured and how AI was used. Knowledge outcomes did better than skills, and AI that augmented student activities did better than AI that replaced them. Thirty-three studies reported large effects only because intervention and control groups did cognitively different activities, so the comparisons were not like for like.

The authors meta-analyzed externally assessed cognitive outcomes and narratively synthesized learner challenges and supportive interventions where pooling was not feasible. Two pairs of raters independently screened and coded the studies, which had to be peer-reviewed, quantitative, and include a comparison group.

The moderator analysis found that the type of learning outcome and the interactive-constructive-active-passive level of the intervention versus the control explained parts of the variation. A coincidence analysis found no combination of the two that was enough on its own to produce large effects.

The review reports that evidence on learner challenges and effective interventions was limited and inconsistently reported. Studies systematically underreported learner-level variables such as AI literacy and metacognitive skills, and process-level factors such as task delegation, prompt quality and verification behaviours.