70/100

Moderate evidence

Cite
More
EmailLinkedInX

Chatbot users found university information 38% faster and more accurately in a 299-person experiment

Study title: AI-Powered Conversational Interfaces in Higher Education: An Experimental Comparison of Chatbots and Traditional Websites for Information Seeking

Evidence type
Randomized controlled trial
Sample
n = 299
Sources
1 sourcePeer-reviewed
Published
1 min read

Study title: AI-Powered Conversational Interfaces in Higher Education: An Experimental Comparison of Chatbots and Traditional Websites for Information Seeking

A randomized experiment with 299 young adult participants compared an AI-powered chatbot with a traditional university website for finding academic and institutional information. Chatbot users finished the tasks in about 38% less time and answered 95.6% of tasks accurately, against 78.5% for website users, a gap of 17.1 percentage points.

Read the study abstract

As artificial intelligence (AI) continues to reshape higher education, traditional web-based methods of providing interested parties with academic and institutional information are increasingly being challenged. The current study provides empirical evidence surrounding young adult perceptions, information-seeking efficiency, and objective task performance when using AI-powered chatbots compared to traditional university websites. Guided by the Technology Readiness and Acceptance Model (TRAM), 299 participants we…

The gain

In an experiment with 299 randomly assigned participants, those using an AI chatbot to find university information finished tasks in about 38% less time and scored 95.6% accuracy versus 78.5% for website users.

In brief

Chatbot users also rated the tool higher on perceived ease of use, perceived usefulness and attitude toward the technology (all p < 0.001). The authors advise universities to treat conversational AI as a complement to existing support infrastructure and to keep evaluating both user perceptions and objective performance.

Main points

  1. The study randomly assigned 299 participants to either a university website or an AI chatbot for information-seeking tasks.
  2. Chatbot users completed the tasks in approximately 38% less time.
  3. Task accuracy was 95.6% for chatbot users and 78.5% for website users, a difference of 17.1 percentage points.
  4. Chatbot users reported significantly higher perceived ease of use, usefulness and attitudes toward the technology (p < 0.001 for each).
  5. The design drew on the Technology Readiness and Acceptance Model (TRAM).

The full story

The experiment, published in Education Sciences, tested whether AI-powered conversational interfaces improve how prospective students and other interested parties find academic and institutional information. Participants were young adults randomly assigned to one of two conditions: a traditional university website or an AI chatbot.

Beyond self-reported perceptions, the study measured objective performance. Chatbot participants were faster by roughly 38% and more accurate, with 95.6% overall task accuracy compared with 78.5% on the website.

The authors conclude that conversational AI appears capable of improving both the experience and the speed and accuracy of locating institutional information. They recommend universities adopt it as complementary support infrastructure while continuing to evaluate perceptions and performance.

Sources

1 on file
  1. Education Sciencesdoi:10.3390/educsci16111966
    Peer-reviewed2026-11-17

Talk it over

Three questions to start on, two more if you have time
Chatbot users finished in about 38% less time and scored 95.6% accuracy against 78.5% for website users. Which of those two gaps would matter more to a university deciding whether to adopt a chatbot, and why?

Start with a question

Chatbot users found university information 38% faster and more accurately in a 299-person experimenthttps://truvace.com/item/ai-powered-conversational-interfaces-in-higher-education-an-experimental-compari

  1. Chatbot users finished in about 38% less time and scored 95.6% accuracy against 78.5% for website users. Which of those two gaps would matter more to a university deciding whether to adopt a chatbot, and why?

    Answer in the thread
  2. The 299 participants were young adults randomly assigned to a university website or an AI chatbot. How might that sample and those tasks differ from the people and questions a real university faces?

    Answer in the thread
  3. The authors advise treating conversational AI as a complement to existing support infrastructure. Who should decide which questions go to the chatbot and which stay with the website or staff?

    Answer in the thread
If you have more time
  • Think of a university website you know and a task a prospective student might do on it. How would you measure time and accuracy for that task the way the researchers did?

    Answer in the thread
  • Chatbot users rated ease of use, usefulness and attitude higher (p < 0.001 for each). What result on accuracy or time would make those ratings matter less to you?

    Answer in the thread

What readers say