TRV-2026-2995Version 1 · Published

Saved · see the current record

Why this version exists

Certified into the record

The exact text we saved

This is the version exactly as it was saved, character for character. The SHA-256 hash below is a 64-character code worked out from that text. Run the same text through SHA-256 and you get the same code; change a single letter and the code comes out completely different. Each version also stores the previous version’s hash, which chains them together so no version can be swapped out quietly.

TRUVACE RECORD VERSION
record: TRV-2026-2995
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-17T12:02:21.053813Z
status: published
lens: g_space
sector: education
headline: Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education
dek: Abstract Assessment feedback in higher education provides fine-grained evidence about students’ specific errors but remains underutilized as a data source for teaching improvement. This study proposes a method based on a large language model (LLM) that transforms unstructured feedback into curriculum-aligned error analysis in three steps, deriving an error taxonomy from the feedback, mapping each piece to curriculum content by retrieval-augmented generation, and aggregating the annotations into error patterns wi…
gain_title: In a study of 6,072 marker comments from four engineering cohorts (2022–2025), an AI pipeline using a taxonomy built from the feedback itself agreed with a human reviewer on 89.1% of error labels, versus 74.7% with an expert-designed taxonomy.
problem_title: (none)
trace_subject: (none)
gain_reading: In a study of 6,072 marker comments from four engineering cohorts (2022–2025), an AI pipeline using a taxonomy built from the feedback itself agreed with a human reviewer on 89.1% of error labels, versus 74.7% with an expert-designed taxonomy.
gain_evidence: The resulting annotations achieved 89.1% agreement with human judgment, substantially higher than the 74.7% obtained when an expert-designed taxonomy was applied to the same data.
problem_reading: (none)
problem_evidence: (none)
quick_read: Researchers at Monash University, Deakin University and the University of Melbourne built a three-step language-model pipeline that turns written assessment feedback into maps of student errors tied to specific lecture content. They applied it to 6,072 pieces of feedback on group design reports from 322 student groups in a third-year undergraduate engineering course, across four cohorts from 2022 to 2025.

Labels produced with a taxonomy generated from the feedback itself agreed with a human reviewer 89.1% of the time (Cohen's kappa 0.91), against 74.7% (kappa 0.64) for a taxonomy designed in advance by the unit coordinator. The pipeline found that missing content and justification made up 66.5% of the 1,255 error-bearing comments in the 2022 data.

Three members of the teaching team reviewed the four-year output in a focus group. They said the patterns matched their marking experience, and the unit coordinator proposed more worked examples and shifting teaching time toward topics with persistent errors. They also said the output should serve as a reference and not as a measure of teaching quality. The work covers one course at one institution.
limitation: The case study covered a single engineering course at one institution with group-based project reports, and the taxonomy was generated from one year of data; its performance in other disciplines, assessment formats and institutions remains untested.
tag: Evidence-backed gain
key_points: The pipeline generated an error taxonomy from 1,738 comments in the 2022 cohort; 258 behavioural descriptions were grouped into five final categories after one human review. | Each comment was linked to one of 374 lecture-slide knowledge units; the 1,255 error-bearing comments mapped to 200 of those units across 39 of 40 modules. | Incomplete or missing content and justification accounted for 66.5% of 2022 errors, 73.3% in Stage A and 54.2% in Stage B, while procedural flaws rose from 13.2% to 30.3% between stages. | Across 2022–2025 the pipeline found 46 error-pattern groups in Stage A and 36 in Stage B; 27 and 21 respectively appeared in two or more years. | The most prevalent 2022 pattern, missing cross-section data and labelling in drawings, fell from 93.9% of groups to 70.3% by 2025, while missing justification for curve coordination rose from 67.3% to 81.1%. | With the expert taxonomy, the Careless and Procedural categories reached only 21.4% and 12.1% agreement, and 102 records (5.9%) belonged to a presentation category the expert taxonomy lacked. | With the data-driven taxonomy, procedural and process flaws were the weakest category at 65.8% agreement; four of five categories exceeded 90%.
rundown: The course has 200 to 300 students a semester, working in groups of five or six on a design project assessed through two written reports. Five to seven teaching assistants marked against rubrics after calibration sessions. The pipeline used OpenAI language models to label feedback, an embedding search to retrieve five candidate knowledge units per comment, and a model-written reasoning note for each annotation.

For validation, a stronger model reviewed each annotation five times and about 90% were confirmed under each taxonomy. The 166 flagged records under the data-driven taxonomy and 187 under the expert one, plus samples of unflagged records, went to an experienced teaching assistant who did not see the pipeline's labels. Knowledge-unit relevance agreement was 92.0% for the data-driven taxonomy and 89.7% for the expert one.

In the focus group, one teaching assistant said he had not expected the errors to be the mainstream issue, and the unit coordinator cited that only 25% of students produced complete cross-section drawings. She proposed benchmark examples, in-class demonstrations and weekly alerts, and said economic evaluation may need two weeks instead of one.

The team also set limits. They said written materials miss oral instruction and help-desk support, and that an improvement such as the 2025 adoption of computer-aided design software may not reflect deeper learning. They said the output cannot be used for teaching evaluation.
sources:
- peer_reviewed | International Journal of Educational Technology in Higher Education | https://doi.org/10.1186/s41239-026-00640-7 | 2026-11-15
prev: 0000000000000000000000000000000000000000000000000000000000000000
SHA-256 hash
e8e975807401737432e1a04ef8f73b2dc846f31117e83257cc9d98034ea4bd1f
Previous version’s hash
0000000000000000000000000000000000000000000000000000000000000000
Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2995 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.