70/100

Moderate evidence

Cite
More
EmailLinkedInX

A study of 6,072 marker comments finds AI matched a human reviewer 89.1% of the time

Study title: Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education

Evidence type
Controlled or validated comparison
Sample
n = 6,072
Sources
1 sourcePeer-reviewed
Published
3 min read

Study title: Curriculum-aligned analysis of assessment feedback: a retrieval-augmented large language model approach for revealing fine-grained student error patterns in higher education

Researchers at Monash University, Deakin University and the University of Melbourne built a three-step language-model pipeline that turns written assessment feedback into maps of student errors tied to specific lecture content. They applied it to 6,072 pieces of feedback on group design reports from 322 student groups in a third-year undergraduate engineering course, across four cohorts from 2022 to 2025.

Read the study abstract

Abstract Assessment feedback in higher education provides fine-grained evidence about students’ specific errors but remains underutilized as a data source for teaching improvement. This study proposes a method based on a large language model (LLM) that transforms unstructured feedback into curriculum-aligned error analysis in three steps, deriving an error taxonomy from the feedback, mapping each piece to curriculum content by retrieval-augmented generation, and aggregating the annotations into error patterns wi…

The gain

In a study of 6,072 marker comments from four engineering cohorts (2022–2025), an AI pipeline using a taxonomy built from the feedback itself agreed with a human reviewer on 89.1% of error labels, versus 74.7% with an expert-designed taxonomy.

In brief

Labels produced with a taxonomy generated from the feedback itself agreed with a human reviewer 89.1% of the time (Cohen's kappa 0.91), against 74.7% (kappa 0.64) for a taxonomy designed in advance by the unit coordinator. The pipeline found that missing content and justification made up 66.5% of the 1,255 error-bearing comments in the 2022 data.

Three members of the teaching team reviewed the four-year output in a focus group. They said the patterns matched their marking experience, and the unit coordinator proposed more worked examples and shifting teaching time toward topics with persistent errors. They also said the output should serve as a reference and not as a measure of teaching quality. The work covers one course at one institution.

Main points

  1. The pipeline generated an error taxonomy from 1,738 comments in the 2022 cohort; 258 behavioural descriptions were grouped into five final categories after one human review.
  2. Each comment was linked to one of 374 lecture-slide knowledge units; the 1,255 error-bearing comments mapped to 200 of those units across 39 of 40 modules.
  3. Incomplete or missing content and justification accounted for 66.5% of 2022 errors, 73.3% in Stage A and 54.2% in Stage B, while procedural flaws rose from 13.2% to 30.3% between stages.
  4. Across 2022–2025 the pipeline found 46 error-pattern groups in Stage A and 36 in Stage B; 27 and 21 respectively appeared in two or more years.
  5. The most prevalent 2022 pattern, missing cross-section data and labelling in drawings, fell from 93.9% of groups to 70.3% by 2025, while missing justification for curve coordination rose from 67.3% to 81.1%.
  6. With the expert taxonomy, the Careless and Procedural categories reached only 21.4% and 12.1% agreement, and 102 records (5.9%) belonged to a presentation category the expert taxonomy lacked.
  7. With the data-driven taxonomy, procedural and process flaws were the weakest category at 65.8% agreement; four of five categories exceeded 90%.

The full story

The course has 200 to 300 students a semester, working in groups of five or six on a design project assessed through two written reports. Five to seven teaching assistants marked against rubrics after calibration sessions. The pipeline used OpenAI language models to label feedback, an embedding search to retrieve five candidate knowledge units per comment, and a model-written reasoning note for each annotation.

For validation, a stronger model reviewed each annotation five times and about 90% were confirmed under each taxonomy. The 166 flagged records under the data-driven taxonomy and 187 under the expert one, plus samples of unflagged records, went to an experienced teaching assistant who did not see the pipeline's labels. Knowledge-unit relevance agreement was 92.0% for the data-driven taxonomy and 89.7% for the expert one.

In the focus group, one teaching assistant said he had not expected the errors to be the mainstream issue, and the unit coordinator cited that only 25% of students produced complete cross-section drawings. She proposed benchmark examples, in-class demonstrations and weekly alerts, and said economic evaluation may need two weeks instead of one.

The team also set limits. They said written materials miss oral instruction and help-desk support, and that an improvement such as the 2025 adoption of computer-aided design software may not reflect deeper learning. They said the output cannot be used for teaching evaluation.

Sources

1 on file
  1. Peer-reviewed2026-11-15

Talk it over

Three questions to start on, two more if you have time
The pipeline agreed with a human reviewer on 89.1% of error labels with a taxonomy built from the feedback, against 74.7% with the unit coordinator's taxonomy. How would you weigh that gain against the weaker procedural category at 65.8%?

Start with a question

A study of 6,072 marker comments finds AI matched a human reviewer 89.1% of the timehttps://truvace.com/item/curriculum-aligned-analysis-of-assessment-feedback-a-retrieval-augmented-large-l

  1. The pipeline agreed with a human reviewer on 89.1% of error labels with a taxonomy built from the feedback, against 74.7% with the unit coordinator's taxonomy. How would you weigh that gain against the weaker procedural category at 65.8%?

    Answer in the thread
  2. Only one experienced teaching assistant, who did not see the pipeline's labels, checked the flagged records. How much would you trust the 89.1% agreement figure, given that a single reviewer made those checks?

    Answer in the thread
  3. If you were one of the three teaching team members in the focus group, what would you do with the finding that missing content and justification made up 66.5% of error-bearing comments in 2022?

    Answer in the thread
If you have more time
  • The teaching team said the output should not be used to judge teaching quality. Who, in a university you know, might be tempted to use it that way, and what would stop them?

    Answer in the thread
  • The study covered one third-year engineering course with 322 student groups. What would you want to see from another subject or another university before trusting the same pipeline there?

    Answer in the thread

What readers say