Truace
Subscribe
70/100

Moderate evidence

Trace: a gain and a problemPublished recordEducationThe Trace

Full story
More
Export record JSONEmailLinkedInX

Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis

Truvace record
TRV-2026-2711raw data (JSON)
Version
v1Published
First published
Evidence type
Systematic review or meta-analysis
Sources
1 sourcePeer-reviewed
sha256
c90cabf7b5d4…check it

Introduction Large language models (LLMs) are increasingly used for scoring and feedback in second-language (L2) assessment, yet the validity of the resulting interpretations remains contested. This review evaluated when LLM-mediated assessment is psychometrically and educationally defensible using an argument-based validity framework focused on construct representation, reliability and reproducibility, fairness, and washback. Methods Following PRISMA 2020, we systematically searched nine databases and repositor…

60Gain side

How the two sides scoreClose call

59Problem side

The gain

In a meta-analysis of 14 studies, AI chatbot feedback on second-language writing produced a small-to-moderate improvement in revision quality and short-term writing outcomes (Hedges' g = 0.42, 95% CI 0.27 to 0.57).

The problem

A systematic review of 52 studies of AI scoring and feedback for second-language learners found that learners sometimes accepted revisions without critical evaluation, and that fairness, reproducibility and construct evidence was too thin to support autonomous high-stakes scoring.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

What this doesn’t fix

Evidence was concentrated in short-term, university-level English-as-a-foreign-language writing studies, with small samples and incomplete reporting of prompts and model versions; some models tested have since been updated or deprecated.

Sources

1 on file
  1. Frontiers in Research Metrics and Analyticsdoi:10.3389/frma.2026.1908592
    Peer-reviewed2026-09-11

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Association between human and LLM scores combining Pearson and Spearman correlations (sensitivity analysis)

    Correlation 0.64 (95% CI 0.47 to 0.77)

    5 studies · search to December 2025

    The review’s own words
    The resulting pooled association estimate was approximately 0.64 (95% CI [0.47, 0.77]; I 2 = 87.7%; t -based prediction interval approximately [−0.20, 0.94]).

  • Agreement between human and LLM scores in second-language assessment (Pearson correlation)

    Correlation (r) 0.66 (95% CI 0.53 to 0.75)

    search to December 2025

    The review’s own words
    Three source-verified L2 studies yielded moderate human-LLM score correspondence (r = 0.66, 95% CI [0.53, 0.75]) with substantial heterogeneity (I 2 = 70.8%).

  • Revision quality and writing improvement with formative LLM feedback, after trim-and-fill adjustment

    Hedges' g 0.39 (95% CI 0.26 to 0.53)

    14 studies · search to December 2025

    The review’s own words
    However, trim-and-fill analysis did not impute any missing studies, and the adjusted pooled effect remained substantively similar to the observed estimate ( g = 0.39, 95% CI [0.26, 0.53]).

  • Revision quality and short-term writing outcomes with LLM-mediated formative feedback

    Hedges' g 0.42 (95% CI 0.27 to 0.57)

    14 studies · search to December 2025

    The review’s own words
    Across 14 studies, LLM-mediated formative feedback produced a small-to-moderate positive effect on revision quality and short-term writing outcomes (g = 0.42, 95% CI [0.27, 0.57]), although heterogeneity was substantial (I 2 = 69.8%) and concerns about cognitive offloading, learner agency, and reproducibility persisted.

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

Cite this record

v1

Truvace Impact Record TRV-2026-2711, v1: “Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis.” Truvace, 2026-11-12. /record/TRV-2026-2711 (accessed on the day you copy it). sha256 c90cabf7b5d4e9b3…

Change history

1 version

Every change made to this record since it was first published, and why. Nothing has changed since it went up.

  1. Publishedv1c90cabf7b5d4…

    Certified into the record

Check this record hasn’t been altered

Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2711 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.