Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2243
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-24T18:34:54.183860Z
status: published
lens: trace
sector: science
headline: Can large language models code meaning-making? A multi-model validation of automated autobiographical reasoning scoring in Chinese adolescent narratives
dek: Autobiographical reasoning (AR) involves drawing self-relevant meaning from personal experiences. It is central to narrative identity research, but manual coding requires substantial time and trained coders. Large language models (LLMs) have shown promise for automating other narrative coding tasks, but their capacity for meaning-making judgments that may depend on implicit event-self connections remains untested. We collected autobiographical narratives from 437 Chinese high school students and evaluated three…
gain_title: In texts with moderate-to-high autobiographical reasoning, majority-vote LLM scoring reached 94.4% accuracy, suggesting partial usefulness for automated coding.
problem_title: Agreement with manual coding was only moderate at best (Cohen's kappa .416), and models missed low-AR texts and over-read emotional expression, so they cannot replace formal manual coding.
trace_subject: LLM scoring of autobiographical reasoning in Chinese high school students' narratives
gain_reading: In texts with moderate-to-high autobiographical reasoning, majority-vote LLM scoring reached 94.4% accuracy, suggesting partial usefulness for automated coding.
gain_evidence: three-run majority-vote accuracy rose from 54.9% for AR-absent texts to 94.4% for moderate-to-high-AR texts
problem_reading: Agreement with manual coding was only moderate at best (Cohen's kappa .416), and models missed low-AR texts and over-read emotional expression, so they cannot replace formal manual coding.
problem_evidence: The best-performing condition (DeepSeek-V3, zero-shot) achieved Run 1 Cohen's κ = .416 | general-purpose LLMs cannot replace formal manual AR coding
quick_read: Researchers gathered autobiographical narratives from 437 Chinese high school students and tested whether three large language models could score them for autobiographical reasoning, meaning drawn from personal experiences. They compared four prompting strategies and three scoring formats against manual coding.

The finding matters because manual coding is slow and needs trained coders, so automation is attractive for narrative identity research. Results show that performance depends heavily on the text, and under the tested conditions the models could not replace human coders. It is unclear whether other models, prompts, or corpora would change this.
limitation: The conclusion applies only to the specific models, prompts, corpus and scoring rules tested, so it may not generalize to other models, languages, or datasets.
tag: Dual reading
key_points: The sample was 437 Chinese high school students' autobiographical narratives, scored by GPT-4o, DeepSeek-V3 and Qwen-Max. | The best condition, DeepSeek-V3 zero-shot, reached a Run 1 Cohen's kappa of .416. | Majority-vote accuracy rose from 54.9% for AR-absent texts to 94.4% for moderate-to-high-AR texts. | Chain-of-thought prompting showed no reliable advantage over zero-shot, as confidence intervals included zero for all three models. | Errors included false negatives concentrated in low-AR texts (OR = 3.99) and false positives linked to emotional expression.
rundown: The study tested three prompting-strategy families across zero-shot, chain-of-thought, 3-shot and 5-shot setups, and binary, ordinal and continuous scoring granularities, to see whether LLMs can judge implicit links between events and the self.

In the fully crossed subset, paired-bootstrap comparisons of chain-of-thought versus zero-shot produced confidence intervals that included zero for every model, so extra reasoning prompts did not clearly help.

Error analysis found two bias patterns: the models tended to miss meaning-making in low-AR texts, and they over-interpreted emotional expression as reasoning. The authors conclude general-purpose models cannot yet replace formal manual coding.
sources:
- peer_reviewed | Behavior Research Methods | https://doi.org/10.3758/s13428-026-03212-9 | 2026-10-23
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

75b43761abf9d4ac551538bb29ba2816252d442597024556d45e3fb3da5dd16e

previous

0000000000000000000000000000000000000000000000000000000000000000