Truace
Subscribe

Researchers tested whether AI chatbots can score life-story writing by 437 Chinese high school students

Study title: Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives

Researchers gathered autobiographical narratives from 437 Chinese high school students and tested whether three large language models could score them for autobiographical reasoning, meaning drawn from personal experiences. They compared four prompting strategies and three scoring formats against manual coding.

Read the study abstract

Autobiographical reasoning (AR) involves drawing self-relevant meaning from personal experiences. It is central to narrative identity research, but manual coding requires substantial time and trained coders. Large language models (LLMs) have shown promise for automating other narrative coding tasks, but their capacity for meaning-making judgments that may depend on implicit event-self connections remains untested. We collected autobiographical narratives from 437 Chinese high school students and evaluated three…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-2243· cite & verify
Peer-reviewed study: Behavior Research Methods
Trace impact readingContested
P 66The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

The finding matters because manual coding is slow and needs trained coders, so automation is attractive for narrative identity research. Results show that performance depends heavily on the text, and under the tested conditions the models could not replace human coders. It is unclear whether other models, prompts, or corpora would change this.

Main points

  1. The sample was 437 Chinese high school students' autobiographical narratives, scored by GPT-4o, DeepSeek-V3 and Qwen-Max.
  2. The best condition, DeepSeek-V3 zero-shot, reached a Run 1 Cohen's kappa of .416.
  3. Majority-vote accuracy rose from 54.9% for AR-absent texts to 94.4% for moderate-to-high-AR texts.
  4. Chain-of-thought prompting showed no reliable advantage over zero-shot, as confidence intervals included zero for all three models.
  5. Errors included false negatives concentrated in low-AR texts (OR = 3.99) and false positives linked to emotional expression.

The gain

In texts with moderate-to-high autobiographical reasoning, majority-vote LLM scoring reached 94.4% accuracy, suggesting partial usefulness for automated coding.

The problem

Agreement with manual coding was only moderate at best (Cohen's kappa .416), and models missed low-AR texts and over-read emotional expression, so they cannot replace formal manual coding.

The rundown

The study tested three prompting-strategy families across zero-shot, chain-of-thought, 3-shot and 5-shot setups, and binary, ordinal and continuous scoring granularities, to see whether LLMs can judge implicit links between events and the self.

In the fully crossed subset, paired-bootstrap comparisons of chain-of-thought versus zero-shot produced confidence intervals that included zero for every model, so extra reasoning prompts did not clearly help.

Error analysis found two bias patterns: the models tended to miss meaning-making in low-AR texts, and they over-interpreted emotional expression as reasoning. The authors conclude general-purpose models cannot yet replace formal manual coding.

What this doesn’t fix

The conclusion applies only to the specific models, prompts, corpus and scoring rules tested, so it may not generalize to other models, languages, or datasets.

Sources

  1. Peer-reviewedBehavior Research Methods2026-10-23

The debate