Moderate evidence
More
Student chat logs with an AI tutor exposed class-wide gaps in a microbiology unit
Study title: Beyond the tool: what a course-aligned AI assistant reveals about student learning in microbiology
- Evidence type
- Observational study
- Sample
- n = 1,300
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-3034cite or check it
Study title: Beyond the tool: what a course-aligned AI assistant reveals about student learning in microbiology
Teachers of a second-year medical microbiology and immunology unit at the University of Sydney, with 500 to 800 students, gave students a course-aligned AI assistant built on the Cogniti platform. It offered scenario-based short-answer practice with feedback tied to a rubric.
Read the study abstract
Course-aligned artificial intelligence (AI) tutors are increasingly offered as learning support in higher education. Beyond providing practice and feedback, they may also help teachers identify gaps in understanding across a cohort. In a second-year medical microbiology and immunology unit of 500 to 800 students, a course-aligned AI assistant built on The University of Sydney’s Cogniti platform gave students scenario-based short-answer practice with rubric-referenced feedback. Its conversation logs turned out to…
The gain
In a University of Sydney microbiology and immunology unit of 500 to 800 students, logs from more than 1300 conversations with a Cogniti-based AI assistant revealed cohort-level knowledge gaps and recurring terminology errors mid-semester, which teachers fed back to the class.
In brief
Across more than 1300 conversations, the logs showed knowledge gaps across the cohort and recurring terminology errors partway through the semester. Teachers used the findings to build an interactive revision resource for lectures.
An exploratory analysis found small positive correlations between practice-oriented use of the assistant and final short-answer marks, though the authors say this should be interpreted cautiously.
Main points
- The assistant was built on The University of Sydney's Cogniti platform and aligned to the course.
- Students practised scenario-based short-answer questions and received rubric-referenced feedback.
- More than 1300 conversations were analysed, revealing cohort-level gaps and recurring terminology errors mid-semester.
- Exploratory analysis found small positive correlations between practice-oriented use and final short-answer marks.
- Insights could be obtained through no-code platform summaries, method-led coding with basic programming, or specialist computational analysis.
- The authors also discuss adapting the approach where no equivalent platform exists.
The full story
The unit enrolled 500 to 800 students. The AI assistant gave them practice scenarios and feedback against a marking rubric, and its conversation logs became a way to see what the whole class found difficult.
Three analysis routes were described, requiring different technical skill: the platform's own no-code summaries, method-led coding with basic programming, and specialist computational analysis. This lets teachers without data expertise still use the logs.
Findings were returned to students through an interactive revision resource used in lectures. The reported link between practice-oriented use and final short-answer marks was small and exploratory.
Sources
1 on file- Microbiology Australiadoi:10.1071/ma26055
Talk it over
Logs from more than 1300 conversations showed class-wide gaps and recurring terminology errors mid-semester. Which students might be missing from those logs, and how could that change what teachers concluded?
Student chat logs with an AI tutor exposed class-wide gaps in a microbiology unithttps://truvace.com/item/beyond-the-tool-what-a-course-aligned-ai-assistant-reveals-about-student-learnin
Logs from more than 1300 conversations showed class-wide gaps and recurring terminology errors mid-semester. Which students might be missing from those logs, and how could that change what teachers concluded?
Answer in the threadThe authors found small positive correlations between practice-oriented use and final short-answer marks, and call them exploratory. What kind of finding would show that the assistant itself helped marks rise?
Answer in the threadTeachers fed the findings back through an interactive revision resource in lectures. If you were one of the 500 to 800 students, how would you feel about teachers reading your conversations with the assistant?
Answer in the thread
If you have more time
The authors describe three analysis routes: platform summaries, method-led coding with basic programming and specialist computational analysis. Which would suit a course you know, and why?
Answer in the threadWhat would a teacher need to tell students about how their chat logs are used before the logs become a teaching tool?
Answer in the thread
ace

What readers say