Moderate evidence
More
Chatbot users found university information 38% faster and more accurately in a 299-person experiment
Study title: AI-Powered Conversational Interfaces in Higher Education: An Experimental Comparison of Chatbots and Traditional Websites for Information Seeking
- Evidence type
- Randomized controlled trial
- Sample
- n = 299
- Sources
- 1 sourcePeer-reviewed
- Published
- 1 min read
- Truvace record
- TRV-2026-3043cite or check it
Study title: AI-Powered Conversational Interfaces in Higher Education: An Experimental Comparison of Chatbots and Traditional Websites for Information Seeking
A randomized experiment with 299 young adult participants compared an AI-powered chatbot with a traditional university website for finding academic and institutional information. Chatbot users finished the tasks in about 38% less time and answered 95.6% of tasks accurately, against 78.5% for website users, a gap of 17.1 percentage points.
Read the study abstract
As artificial intelligence (AI) continues to reshape higher education, traditional web-based methods of providing interested parties with academic and institutional information are increasingly being challenged. The current study provides empirical evidence surrounding young adult perceptions, information-seeking efficiency, and objective task performance when using AI-powered chatbots compared to traditional university websites. Guided by the Technology Readiness and Acceptance Model (TRAM), 299 participants we…
The gain
In an experiment with 299 randomly assigned participants, those using an AI chatbot to find university information finished tasks in about 38% less time and scored 95.6% accuracy versus 78.5% for website users.
In brief
Chatbot users also rated the tool higher on perceived ease of use, perceived usefulness and attitude toward the technology (all p < 0.001). The authors advise universities to treat conversational AI as a complement to existing support infrastructure and to keep evaluating both user perceptions and objective performance.
Main points
- The study randomly assigned 299 participants to either a university website or an AI chatbot for information-seeking tasks.
- Chatbot users completed the tasks in approximately 38% less time.
- Task accuracy was 95.6% for chatbot users and 78.5% for website users, a difference of 17.1 percentage points.
- Chatbot users reported significantly higher perceived ease of use, usefulness and attitudes toward the technology (p < 0.001 for each).
- The design drew on the Technology Readiness and Acceptance Model (TRAM).
The full story
The experiment, published in Education Sciences, tested whether AI-powered conversational interfaces improve how prospective students and other interested parties find academic and institutional information. Participants were young adults randomly assigned to one of two conditions: a traditional university website or an AI chatbot.
Beyond self-reported perceptions, the study measured objective performance. Chatbot participants were faster by roughly 38% and more accurate, with 95.6% overall task accuracy compared with 78.5% on the website.
The authors conclude that conversational AI appears capable of improving both the experience and the speed and accuracy of locating institutional information. They recommend universities adopt it as complementary support infrastructure while continuing to evaluate perceptions and performance.
Sources
1 on file- Education Sciencesdoi:10.3390/educsci16111966
Talk it over
Chatbot users finished in about 38% less time and scored 95.6% accuracy against 78.5% for website users. Which of those two gaps would matter more to a university deciding whether to adopt a chatbot, and why?
Chatbot users found university information 38% faster and more accurately in a 299-person experimenthttps://truvace.com/item/ai-powered-conversational-interfaces-in-higher-education-an-experimental-compari
Chatbot users finished in about 38% less time and scored 95.6% accuracy against 78.5% for website users. Which of those two gaps would matter more to a university deciding whether to adopt a chatbot, and why?
Answer in the threadThe 299 participants were young adults randomly assigned to a university website or an AI chatbot. How might that sample and those tasks differ from the people and questions a real university faces?
Answer in the threadThe authors advise treating conversational AI as a complement to existing support infrastructure. Who should decide which questions go to the chatbot and which stay with the website or staff?
Answer in the thread
If you have more time
Think of a university website you know and a task a prospective student might do on it. How would you measure time and accuracy for that task the way the researchers did?
Answer in the threadChatbot users rated ease of use, usefulness and attitude higher (p < 0.001 for each). What result on accuracy or time would make those ratings matter less to you?
Answer in the thread
ace
What readers say