Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 4 studies (420 students) links AI tutoring to higher engineering scores, but evidence is weak

Study title: AI-based personalized learning and academic performance in computing and engineering higher education: a systematic review and meta-analysis of controlled studies

Evidence type
Systematic review or meta-analysis
Sample
n = 420
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2730cite or check it

Study title: AI-based personalized learning and academic performance in computing and engineering higher education: a systematic review and meta-analysis of controlled studies

A systematic review and meta-analysis in Frontiers in Education pooled four controlled studies with 420 students (208 in AI groups, 212 in comparison groups), all in computing and engineering courses in Egypt, China and Morocco. The pooled effect was a Hedges' g of 1.01 (95% CI 0.33 to 1.68) favouring AI-based personalized learning on objectively measured performance.

Read the study abstract

Introduction The rapid adoption of Artificial Intelligence (AI) tools for personalized learning in higher education has generated growing interest in their effects on student outcomes, yet comparative evidence remains limited. This systematic review and meta-analysis examined whether AI-based personalized learning is associated with differences in objectively measured academic performance, compared with instruction without the AI component, among higher education students. Methods Following PRISMA 2020, Scopus a…

64Gain side

How the two sides scoreClose call

66Problem side

The gain

A meta-analysis of four controlled studies of 420 computing and engineering students found AI-based personalized learning was associated with higher post-course performance than instruction without AI (pooled Hedges' g 1.01, 95% CI 0.33 to 1.68).

The problem

The same four-study meta-analysis of 420 students rated the evidence very low certainty: with small-sample adjustment the interval (-0.10 to 2.11) included no effect, heterogeneity was I2 = 89%, and two of four studies were at serious risk of bias.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

The headline figure is shaky. With a small-sample adjustment the interval ran from -0.10 to 2.11 and included no effect, and the prediction interval (-0.42 to 2.43) also included zero. Study estimates ranged from g = 0.20 to 1.60, and the authors rated the certainty of evidence very low.

The authors say the pooled figure is likely an upper bound and that causal conclusions are not warranted. Only one study was cluster-randomized, and several tested AI bundled with other teaching changes, so the effect describes an instructional package rather than AI alone.

Main points

  1. Searches of Scopus and Web of Science in April 2026 found 84 unique records; four controlled studies met the criteria after full-text review of 11.
  2. The studies came from Egypt, China and Morocco and covered C# programming, software engineering, introductory thermodynamics and a graduate ocean-engineering course.
  3. Three interventions used ChatGPT-based tools and one used a knowledge-graph precision-teaching platform.
  4. Individual effect sizes ran from g = 0.20 (cluster-randomized study) to g = 1.60, with heterogeneity of I2 = 89%.
  5. Leave-one-out estimates ranged from 0.80 to 1.22; excluding the two studies at serious risk of bias gave g = 0.91 with a Wald interval of -0.40 to 2.22.
  6. Two studies, carrying 53% of the weight, were at serious risk of bias; one reported an effect size (d = 0.85) inconsistent with its own data, which imply about 1.6.
  7. The pooled estimate exceeds earlier tutoring-system benchmarks of 0.66 and 0.76, which the authors attribute partly to small samples and researcher-developed tests.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Post-intervention academic performance of higher education computing and engineering students with AI-based personalized learning versus instruction without AI

    Hedges' g 1.01 (95% CI 0.33 to 1.68)

    420 people · search to April 2026

    The review’s own words
    The pooled estimate was g = 1.01 (95% CI 0.33–1.68; z = 2.93, p = .003).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The review followed PRISMA 2020 and included studies where an AI system adapted content, feedback, pacing or recommendations and where a concurrent or historical comparison group existed. Outcomes had to be objective, such as tests, exams or rubric-scored work. Seven full-text reports were excluded, mainly for having no comparison group or no objective outcome.

The included designs varied widely. One study randomly assigned 27 student groups (13 AI, 14 control) to ChatGPT feedback versus instructor feedback and found a small, imprecise difference (g = 0.20). Another allocated a convenience sample to ChatGPT use after three hours of prompt-writing training. A third compared a 2023-2024 cohort with a 2022-2023 historical cohort.

Sensitivity checks kept the direction positive, with pooled estimates between about 0.8 and 1.2. Under small-sample inference the interval included zero in the primary analysis and in every leave-one-out analysis, and excluded zero only when an alternative composite score or rescaled standard deviation was used.

The authors conclude the evidence does not justify expecting uniform or large gains from adding AI tools to these courses. The largest effects came from studies that embedded AI in a deliberate instructional design, such as structured prompting and regular feedback, and they advise institutions to evaluate such tools locally with objective assessments.

What this doesn’t fix

Only four studies with 420 students were available, only one had a verifiable randomized design, and no intervention lasted longer than one semester, so the pooled result cannot establish causation or apply beyond computing and engineering courses.

Sources

1 on file
  1. Frontiers in Educationdoi:10.3389/feduc.2026.1970736
    Peer-reviewed2026-10-29

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Review of 4 studies (420 students) links AI tutoring to higher engineering scores, but evidence is weakhttps://truvace.com/item/ai-based-personalized-learning-and-academic-performance-in-computing-and-enginee

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain A meta-analysis of four controlled studies of 420 computing and engineering students found AI-based personalized learning was associated with higher post-course performance than instruction without AI (pooled Hedges' g 1.01, 95% CI 0.33 to 1.68).

    The problem The same four-study meta-analysis of 420 students rated the evidence very low certainty: with small-sample adjustment the interval (-0.10 to 2.11) included no effect, heterogeneity was I2 = 89%, and two of four studies were at serious risk of bias.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 420, from Frontiers in Education.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Only four studies with 420 students were available, only one had a verifiable randomized design, and no intervention lasted longer than one semester, so the pooled result cannot establish causation or apply beyond computing and engineering courses.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education