Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

AI feedback modestly improved language learners' revisions, a review of 14 studies finds

Study title: Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis

Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2711cite or check it

Study title: Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis

A systematic review and meta-analysis of 52 empirical studies published from January 2022 to December 2025 examined whether AI chatbot scoring and feedback for second-language learners can be defended on validity grounds. Across 14 studies, formative AI feedback improved revision quality and short-term writing outcomes by g = 0.42 (95% CI 0.27 to 0.57).

Read the study abstract

Introduction Large language models (LLMs) are increasingly used for scoring and feedback in second-language (L2) assessment, yet the validity of the resulting interpretations remains contested. This review evaluated when LLM-mediated assessment is psychometrically and educationally defensible using an argument-based validity framework focused on construct representation, reliability and reproducibility, fairness, and washback. Methods Following PRISMA 2020, we systematically searched nine databases and repositor…

60Gain side

How the two sides scoreClose call

59Problem side

The gain

In a meta-analysis of 14 studies, AI chatbot feedback on second-language writing produced a small-to-moderate improvement in revision quality and short-term writing outcomes (Hedges' g = 0.42, 95% CI 0.27 to 0.57).

The problem

A systematic review of 52 studies of AI scoring and feedback for second-language learners found that learners sometimes accepted revisions without critical evaluation, and that fairness, reproducibility and construct evidence was too thin to support autonomous high-stakes scoring.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Score agreement with human raters was less well established. Only three source-verified studies had directly comparable Pearson correlations, and they pooled to r = 0.66 (95% CI 0.53 to 0.75) with I² of 70.8%. The authors withdrew an earlier 21-study composite estimate of r = 0.74 because it could not be reconstructed from archived materials.

The authors conclude that the evidence supports bounded formative, rubric-guided and human-supervised uses, but not autonomous high-stakes scoring. Fairness was the least studied domain, and most studies were short-term, so effects on long-term learning are unknown.

Main points

  1. The washback meta-analysis covered 14 studies and found g = 0.42 for revision quality under formative conditions, with a prediction interval of 0.05 to 0.78 and I² of 69.8%.
  2. Egger's test showed funnel-plot asymmetry (p < 0.001), but trim-and-fill imputed no missing studies and the adjusted effect was g = 0.39 (95% CI 0.26 to 0.53).
  3. Three source-verified L2 studies pooled to r = 0.66 for human-AI score correspondence; adding two Spearman correlations gave about 0.64 with I² of 87.7%.
  4. Prompt text was absent or only partly reported in 31 of 52 studies (60%), and model version was missing in 14 (27%).
  5. Five studies re-ran identical inputs; score variation ranged from negligible to two or more scale points, with more variability under open-ended prompting.
  6. Nine studies ran subgroup fairness analyses; two found harsher scoring of lower-proficiency learners on discourse-level dimensions, and the median subgroup size was 34.
  7. Learners who used AI feedback extensively over several weeks reported uncertainty about voice and authorship, according to six qualitative studies relying on self-report.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Association between human and LLM scores combining Pearson and Spearman correlations (sensitivity analysis)

    Correlation 0.64 (95% CI 0.47 to 0.77)

    5 studies · search to December 2025

    The review’s own words
    The resulting pooled association estimate was approximately 0.64 (95% CI [0.47, 0.77]; I 2 = 87.7%; t -based prediction interval approximately [−0.20, 0.94]).

  • Agreement between human and LLM scores in second-language assessment (Pearson correlation)

    Correlation (r) 0.66 (95% CI 0.53 to 0.75)

    search to December 2025

    The review’s own words
    Three source-verified L2 studies yielded moderate human-LLM score correspondence (r = 0.66, 95% CI [0.53, 0.75]) with substantial heterogeneity (I 2 = 70.8%).

  • Revision quality and writing improvement with formative LLM feedback, after trim-and-fill adjustment

    Hedges' g 0.39 (95% CI 0.26 to 0.53)

    14 studies · search to December 2025

    The review’s own words
    However, trim-and-fill analysis did not impute any missing studies, and the adjusted pooled effect remained substantively similar to the observed estimate ( g = 0.39, 95% CI [0.26, 0.53]).

  • Revision quality and short-term writing outcomes with LLM-mediated formative feedback

    Hedges' g 0.42 (95% CI 0.27 to 0.57)

    14 studies · search to December 2025

    The review’s own words
    Across 14 studies, LLM-mediated formative feedback produced a small-to-moderate positive effect on revision quality and short-term writing outcomes (g = 0.42, 95% CI [0.27, 0.57]), although heterogeneity was substantial (I 2 = 69.8%) and concerns about cognitive offloading, learner agency, and reproducibility persisted.

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The team searched nine databases and repositories (including Web of Science, Scopus, ERIC and arXiv) in December 2025, finding 3,847 records. After deduplication and screening, 183 were assessed in full text and 52 studies were included; 39 were peer-reviewed and 13 were preprints. Most studies examined university-level English-as-a-foreign-language writing, and median sample size was 67 (range 12 to 412).

LLMs served as automated scorers in 24 studies, formative feedback generators in 18, and revision mediators in 14. Agreement varied by skill: one study reported a Spearman correlation of 0.79 for linguistic accuracy, while another reported Pearson correlations of 0.364 for language and 0.181 for organization. Several studies found AI scores close to experts' while relying on different textual features.

Qualitative work documented benefits such as more engagement with iterative revision and greater metalinguistic awareness when feedback came with guided reflection. It also documented shorter, less reflective revision than under teacher-mediated feedback. The longest intervention lasted 16 weeks, and most ran four to eight weeks.

What this doesn’t fix

Evidence was concentrated in short-term, university-level English-as-a-foreign-language writing studies, with small samples and incomplete reporting of prompts and model versions; some models tested have since been updated or deprecated.

Sources

1 on file
  1. Frontiers in Research Metrics and Analyticsdoi:10.3389/frma.2026.1908592
    Peer-reviewed2026-09-11

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

AI feedback modestly improved language learners' revisions, a review of 14 studies findshttps://truvace.com/item/conditional-validity-in-llm-mediated-l2-assessment-an-argument-based-systematic-

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a meta-analysis of 14 studies, AI chatbot feedback on second-language writing produced a small-to-moderate improvement in revision quality and short-term writing outcomes (Hedges' g = 0.42, 95% CI 0.27 to 0.57).

    The problem A systematic review of 52 studies of AI scoring and feedback for second-language learners found that learners sometimes accepted revisions without critical evaluation, and that fairness, reproducibility and construct evidence was too thin to support autonomous high-stakes scoring.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis, from Frontiers in Research Metrics and Analytics.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Evidence was concentrated in short-term, university-level English-as-a-foreign-language writing studies, with small samples and incomplete reporting of prompts and model versions; some models tested have since been updated or deprecated.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education