The exact text we saved

This is the version exactly as it was saved, character for character. The SHA-256 hash below is a 64-character code worked out from that text. Run the same text through SHA-256 and you get the same code; change a single letter and the code comes out completely different. Each version also stores the previous version’s hash, which chains them together so no version can be swapped out quietly.

TRUVACE RECORD VERSION
record: TRV-2026-2711
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-12T17:13:18.443147Z
status: published
lens: trace
sector: education
headline: Conditional validity in LLM-mediated L2 assessment: an argument-based systematic review and meta-analysis
dek: Introduction Large language models (LLMs) are increasingly used for scoring and feedback in second-language (L2) assessment, yet the validity of the resulting interpretations remains contested. This review evaluated when LLM-mediated assessment is psychometrically and educationally defensible using an argument-based validity framework focused on construct representation, reliability and reproducibility, fairness, and washback. Methods Following PRISMA 2020, we systematically searched nine databases and repositor…
gain_title: In a meta-analysis of 14 studies, AI chatbot feedback on second-language writing produced a small-to-moderate improvement in revision quality and short-term writing outcomes (Hedges' g = 0.42, 95% CI 0.27 to 0.57).
problem_title: A systematic review of 52 studies of AI scoring and feedback for second-language learners found that learners sometimes accepted revisions without critical evaluation, and that fairness, reproducibility and construct evidence was too thin to support autonomous high-stakes scoring.
trace_subject: LLM-mediated feedback and scoring in second-language writing assessment
gain_reading: In a meta-analysis of 14 studies, AI chatbot feedback on second-language writing produced a small-to-moderate improvement in revision quality and short-term writing outcomes (Hedges' g = 0.42, 95% CI 0.27 to 0.57).
gain_evidence: LLM-mediated formative feedback produced a small-to-moderate positive effect on revision quality and short-term writing outcomes ( g = 0.42, 95% CI [0.27, 0.57])
problem_reading: A systematic review of 52 studies of AI scoring and feedback for second-language learners found that learners sometimes accepted revisions without critical evaluation, and that fairness, reproducibility and construct evidence was too thin to support autonomous high-stakes scoring.
problem_evidence: Several studies also reported cognitive offloading, acceptance of revisions without critical evaluation, and reduced metacognitive engagement during revision | does not justify autonomous high-stakes scoring or broad claims of construct validity, fairness, and generalizability
quick_read: A systematic review and meta-analysis of 52 empirical studies published from January 2022 to December 2025 examined whether AI chatbot scoring and feedback for second-language learners can be defended on validity grounds. Across 14 studies, formative AI feedback improved revision quality and short-term writing outcomes by g = 0.42 (95% CI 0.27 to 0.57).

Score agreement with human raters was less well established. Only three source-verified studies had directly comparable Pearson correlations, and they pooled to r = 0.66 (95% CI 0.53 to 0.75) with I² of 70.8%. The authors withdrew an earlier 21-study composite estimate of r = 0.74 because it could not be reconstructed from archived materials.

The authors conclude that the evidence supports bounded formative, rubric-guided and human-supervised uses, but not autonomous high-stakes scoring. Fairness was the least studied domain, and most studies were short-term, so effects on long-term learning are unknown.
limitation: Evidence was concentrated in short-term, university-level English-as-a-foreign-language writing studies, with small samples and incomplete reporting of prompts and model versions; some models tested have since been updated or deprecated.
tag: Dual reading
key_points: The washback meta-analysis covered 14 studies and found g = 0.42 for revision quality under formative conditions, with a prediction interval of 0.05 to 0.78 and I² of 69.8%. | Egger's test showed funnel-plot asymmetry (p < 0.001), but trim-and-fill imputed no missing studies and the adjusted effect was g = 0.39 (95% CI 0.26 to 0.53). | Three source-verified L2 studies pooled to r = 0.66 for human-AI score correspondence; adding two Spearman correlations gave about 0.64 with I² of 87.7%. | Prompt text was absent or only partly reported in 31 of 52 studies (60%), and model version was missing in 14 (27%). | Five studies re-ran identical inputs; score variation ranged from negligible to two or more scale points, with more variability under open-ended prompting. | Nine studies ran subgroup fairness analyses; two found harsher scoring of lower-proficiency learners on discourse-level dimensions, and the median subgroup size was 34. | Learners who used AI feedback extensively over several weeks reported uncertainty about voice and authorship, according to six qualitative studies relying on self-report.
rundown: The team searched nine databases and repositories (including Web of Science, Scopus, ERIC and arXiv) in December 2025, finding 3,847 records. After deduplication and screening, 183 were assessed in full text and 52 studies were included; 39 were peer-reviewed and 13 were preprints. Most studies examined university-level English-as-a-foreign-language writing, and median sample size was 67 (range 12 to 412).

LLMs served as automated scorers in 24 studies, formative feedback generators in 18, and revision mediators in 14. Agreement varied by skill: one study reported a Spearman correlation of 0.79 for linguistic accuracy, while another reported Pearson correlations of 0.364 for language and 0.181 for organization. Several studies found AI scores close to experts' while relying on different textual features.

Qualitative work documented benefits such as more engagement with iterative revision and greater metalinguistic awareness when feedback came with guided reflection. It also documented shorter, less reflective revision than under teacher-mediated feedback. The longest intervention lasted 16 weeks, and most ran four to eight weeks.
sources:
- peer_reviewed | Frontiers in Research Metrics and Analytics | https://doi.org/10.3389/frma.2026.1908592 | 2026-09-11
prev: 0000000000000000000000000000000000000000000000000000000000000000

SHA-256 hash

c90cabf7b5d4e9b3d2b100bcb6d62255cd7f460ec01561c83c8ac76630b7653e

Previous version’s hash

0000000000000000000000000000000000000000000000000000000000000000