Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Meta-analysis of 10 studies finds ChatGPT-assisted teaching lifted engineering and vocational learning

Study title: Estimating the Effect of ChatGPT-Assisted Instruction on Learning Outcomes in TVET, Vocational Education, and Engineering Education: A Meta-Analysis

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2824cite or check it

Study title: Estimating the Effect of ChatGPT-Assisted Instruction on Learning Outcomes in TVET, Vocational Education, and Engineering Education: A Meta-Analysis

A meta-analysis by researchers at Universitas Negeri Padang pooled 10 experimental or quasi-experimental studies from Scopus published 2022–2026. It found that students taught with ChatGPT scored higher than comparison groups on cognitive, affective and psychomotor outcomes, with a pooled Hedges' g of 0.739 (95% CI 0.469–1.010), a moderate effect.

Read the study abstract

Aim/Purpose: The aim of this study is to synthesize empirical evidence from Scopus-indexed studies and estimate the pooled effect of ChatGPT-assisted instruction on learning outcomes across the included TVET, vocational education, and engineering education contexts. Background: The rapid proliferation of ChatGPT has attracted growing interest in its potential to facilitate learning in TVET, vocational education, and engineering education. However, empirical findings remain inconsistent. Some studies report large…

The gain

A meta-analysis of 10 experimental and quasi-experimental studies in vocational, engineering and related courses found ChatGPT-assisted instruction beat comparison teaching, with a moderate pooled effect (Hedges' g = 0.739, 95% CI 0.469–1.010).

In brief

Results varied widely between studies (I² = 83.12%), from g = 0.12 for creative ideation among 51 students in a crossover trial to 1.54 for conceptual understanding in a thermodynamics course of 120 students. Restricting the analysis to directly assessed cognitive, skill and performance outcomes gave 0.689, still significant.

The evidence comes mostly from higher-education programming and engineering courses. Hands-on trade programs, apprenticeships and non-higher-education vocational learners were poorly represented or absent, so the authors say the estimate should not be generalised to all vocational or engineering settings.

Main points

  1. Ten studies were pooled: 4 assessed as randomized trials and 6 as quasi-experiments, with group sizes from 21 to 150 students.
  2. The pooled effect was g = 0.739 (SE 0.138, z = 5.362, p < .001) using a random-effects model.
  3. Heterogeneity was substantial (Q(9) = 49.838, τ² = 0.153, I² = 83.121%), so the figure is an average, not uniform effectiveness.
  4. Results held across assumed within-study correlations: pooled g ranged from 0.730 to 0.753.
  5. Of 22 reported outcomes, 13 (59.1%) were cognitive, 8 (36.4%) affective and 1 (4.5%) psychomotor.
  6. Rosenthal's fail-safe N was 557 and Kendall's τ = 0.250 (p = 0.321) found no clear funnel-plot asymmetry, but with 10 studies these tests had limited power.
  7. Concerns in the primary studies included inaccurate or biased outputs, plagiarism, over-reliance, reduced creativity, privacy and subscription costs.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Learning outcomes (cognitive, affective, psychomotor) with ChatGPT-assisted instruction versus comparison conditions in vocational and engineering education

    Hedges' g 0.739 (95% CI 0.469 to 1.010)

    10 studies

    The review’s own words
    (Hedges’ g = 0.739, 95% CI = 0.469–1.010, p < .001)

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors searched Scopus on April 8, 2026 for ChatGPT studies in TVET, vocational and engineering education, then screened for original experimental or quasi-experimental, English-language, open-access studies. Ten had enough statistics to calculate effect sizes. One author did all screening, and quality was appraised with JBI checklists.

Study-level effects ranged widely. A programming experiment with 45 students gave a composite g of 1.07, a 61-student peer-assessment trial gave 0.99, a cluster-randomized trial of 254 students gave 0.45, and a tourism statistics study of 300 students gave 0.64 for creative thinking. The crossover study of 51 participants gave 0.12.

The included papers named opportunities such as programming support and personalised learning, alongside concerns about over-reliance and accuracy. Proposed strategies, including prompt training and verifying AI answers, were not tested as moderators, so the authors treat them as proposals. They recommend cautious use of ChatGPT as a supplementary tool.

Sources

1 on file
  1. Journal of Information Technology Education: Researchdoi:10.28945/5877
    Peer-reviewed2026-01-01

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

Meta-analysis of 10 studies finds ChatGPT-assisted teaching lifted engineering and vocational learninghttps://truvace.com/item/estimating-the-effect-of-chatgpt-assisted-instruction-on-learning-outcomes-in-tv

  1. Who gets this benefit, and who was left out of the study or story?

    The gain A meta-analysis of 10 experimental and quasi-experimental studies in vocational, engineering and related courses found ChatGPT-assisted instruction beat comparison teaching, with a moderate pooled effect (Hedges' g = 0.739, 95% CI 0.469–1.010).

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from Journal of Information Technology Education: Research.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The review covered only English-language, open-access Scopus studies from 2022 to 2026, so it may have missed studies and the pooled estimate may be higher or lower than the full literature would give.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education