aceEmerging evidence
More
Distance students said chatbot failed on complex requests; 15 interviewed
Study title: Student experiences with an LLM- and RAG-based chatbot for institutional services in Open and Distance Education: a qualitative study
- Evidence type
- Interviews or qualitative study
- Sample
- n = 15
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2508cite or check it
Study title: Student experiences with an LLM- and RAG-based chatbot for institutional services in Open and Distance Education: a qualitative study
A qualitative study at the Faculty of Open and Distance Education, Atatürk University, examined how students experienced an AI chatbot built on a large language model and retrieval-augmented generation for institutional services. Of 25,463 service sessions recorded in December 2025, 25,351 entered the chatbot pathway and were recorded as resolved, while 112 went directly to live support.
Read the study abstract
This study examines Open and Distance Education students’ experiences of institutional services delivered through an LLM- and RAG-based chatbot, focusing on satisfaction, trust, institutional commitment, and reuse intention. At Faculty of Open and Distance Education, Atatürk University, 25,463 service sessions were recorded in December 2025. Of these, 25,351 entered the chatbot pathway and were recorded as resolved, while 112 directly accessed live support. Cross-channel verification identified 349 students whos…
The problem
In interviews with 15 Atatürk University distance-education students, participants said irrelevant chatbot answers made them cautious and that complex, multi-step requests exposed the limits of automation, increasing the need for human handoff.
In brief
Researchers interviewed 24 volunteers and kept 15 transcripts after screening. Thematic analysis produced four themes: service value depended on accurate, actionable guidance; satisfaction and trust grew through successful outcomes and verification; complex requests exposed automation limits; and reuse intention depended on past outcomes and expected problem complexity.
The findings point to context-sensitive guidance, continuity and human support built into the chat interface. With only 15 analysed interviews from one institution, they describe student experience rather than measured service quality.
Main points
- The study covered 25,463 service sessions at Atatürk University's open and distance education faculty in December 2025.
- 25,351 sessions entered the chatbot pathway and were recorded as resolved; 112 went directly to live support.
- Cross-channel verification identified 349 students whose qualifying service episodes stayed within the chatbot-mediated journey for the same issue.
- Of 24 completed interviews, 15 remained after transcript-level screening.
- Students' satisfaction and trust developed through successful outcomes and verification, while irrelevant responses generated caution.
- Complex, context-dependent and multi-step requests increased the relevance of in-interface human handoff.
- Reuse intention depended on prior outcomes and anticipated problem complexity; institutional commitment reflected broader educational and service experiences.
The full story
The study drew on service records from December 2025 at the Faculty of Open and Distance Education, Atatürk University. Of 25,463 sessions, 25,351 entered the chatbot pathway and were recorded as resolved, and 112 accessed live support directly.
For interviews, cross-channel verification identified 349 students whose qualifying episodes remained within the chatbot-mediated journey for the same issue. Recruitment was randomly ordered, and the 24th volunteer was reached on the 52nd sequential recruitment call. Of 24 completed interviews, 15 remained after transcript-level screening.
Thematic analysis generated four higher-order themes. Students valued accurate, actionable guidance that supported resolution, and trust built through successful outcomes and verification. Irrelevant responses made them cautious, and complex or multi-step requests showed where automation fell short.
Students' willingness to reuse the chatbot depended on earlier outcomes and how complex they expected their next problem to be, while commitment to the institution reflected wider educational and service experiences. The authors emphasise context-sensitive guidance, continuity and integrated human support.
Sources
1 on file- Open Learning: The Journal of Open, Distance and e-Learningdoi:10.1080/02680513.2026.2758098
Talk it over
Who carries this cost, and what would have to change to prevent it?
Distance students said chatbot failed on complex requests; 15 interviewedhttps://truvace.com/item/student-experiences-with-an-llm-and-rag-based-chatbot-for-institutional-services
Who carries this cost, and what would have to change to prevent it?
The problem In interviews with 15 Atatürk University distance-education students, participants said irrelevant chatbot answers made them cautious and that complex, multi-step requests exposed the limits of automation, increasing the need for human handoff.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on interviews or another qualitative study with a sample of 15, from Open Learning: The Journal of Open, Distance and e-Learning.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat Findings rest on 15 interviews retained from 24 completed at one faculty, drawn from students whose episodes stayed within the chatbot journey.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say