63/100

Moderate evidence

Cite
More
EmailLinkedInX

Student chat logs with an AI tutor exposed class-wide gaps in a microbiology unit

Study title: Beyond the tool: what a course-aligned AI assistant reveals about student learning in microbiology

Evidence type
Observational study
Sample
n = 1,300
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Beyond the tool: what a course-aligned AI assistant reveals about student learning in microbiology

Teachers of a second-year medical microbiology and immunology unit at the University of Sydney, with 500 to 800 students, gave students a course-aligned AI assistant built on the Cogniti platform. It offered scenario-based short-answer practice with feedback tied to a rubric.

Read the study abstract

Course-aligned artificial intelligence (AI) tutors are increasingly offered as learning support in higher education. Beyond providing practice and feedback, they may also help teachers identify gaps in understanding across a cohort. In a second-year medical microbiology and immunology unit of 500 to 800 students, a course-aligned AI assistant built on The University of Sydney’s Cogniti platform gave students scenario-based short-answer practice with rubric-referenced feedback. Its conversation logs turned out to…

The gain

In a University of Sydney microbiology and immunology unit of 500 to 800 students, logs from more than 1300 conversations with a Cogniti-based AI assistant revealed cohort-level knowledge gaps and recurring terminology errors mid-semester, which teachers fed back to the class.

In brief

Across more than 1300 conversations, the logs showed knowledge gaps across the cohort and recurring terminology errors partway through the semester. Teachers used the findings to build an interactive revision resource for lectures.

An exploratory analysis found small positive correlations between practice-oriented use of the assistant and final short-answer marks, though the authors say this should be interpreted cautiously.

Main points

  1. The assistant was built on The University of Sydney's Cogniti platform and aligned to the course.
  2. Students practised scenario-based short-answer questions and received rubric-referenced feedback.
  3. More than 1300 conversations were analysed, revealing cohort-level gaps and recurring terminology errors mid-semester.
  4. Exploratory analysis found small positive correlations between practice-oriented use and final short-answer marks.
  5. Insights could be obtained through no-code platform summaries, method-led coding with basic programming, or specialist computational analysis.
  6. The authors also discuss adapting the approach where no equivalent platform exists.

The full story

The unit enrolled 500 to 800 students. The AI assistant gave them practice scenarios and feedback against a marking rubric, and its conversation logs became a way to see what the whole class found difficult.

Three analysis routes were described, requiring different technical skill: the platform's own no-code summaries, method-led coding with basic programming, and specialist computational analysis. This lets teachers without data expertise still use the logs.

Findings were returned to students through an interactive revision resource used in lectures. The reported link between practice-oriented use and final short-answer marks was small and exploratory.

Sources

1 on file
  1. Microbiology Australiadoi:10.1071/ma26055
    Peer-reviewed2026-11-18

Talk it over

Three questions to start on, two more if you have time
Logs from more than 1300 conversations showed class-wide gaps and recurring terminology errors mid-semester. Which students might be missing from those logs, and how could that change what teachers concluded?

Start with a question

Student chat logs with an AI tutor exposed class-wide gaps in a microbiology unithttps://truvace.com/item/beyond-the-tool-what-a-course-aligned-ai-assistant-reveals-about-student-learnin

  1. Logs from more than 1300 conversations showed class-wide gaps and recurring terminology errors mid-semester. Which students might be missing from those logs, and how could that change what teachers concluded?

    Answer in the thread
  2. The authors found small positive correlations between practice-oriented use and final short-answer marks, and call them exploratory. What kind of finding would show that the assistant itself helped marks rise?

    Answer in the thread
  3. Teachers fed the findings back through an interactive revision resource in lectures. If you were one of the 500 to 800 students, how would you feel about teachers reading your conversations with the assistant?

    Answer in the thread
If you have more time
  • The authors describe three analysis routes: platform summaries, method-led coding with basic programming and specialist computational analysis. Which would suit a course you know, and why?

    Answer in the thread
  • What would a teacher need to tell students about how their chat logs are used before the logs become a teaching tool?

    Answer in the thread

What readers say