Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Texts from a course chatbot raised A and B grades by 4 points in two randomized trials

Study title: Let’s Chat: Leveraging Chatbot Outreach for Improved Course Performance

Evidence type
Randomized controlled trial
Sample
n = 2,483
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2480cite or check it

Study title: Let’s Chat: Leveraging Chatbot Outreach for Improved Course Performance

Researchers ran randomized controlled trials of a text-message chatbot with non-generative AI in two large online courses at Georgia State University: Introduction to American Government and Principles of Microeconomics. Students assigned to the chatbot were four percentage points more likely to earn an A or B than the 61% of control students who did, about a 7% relative increase.

Read the study abstract

This study provides pre-registered, experimental evidence on the use of non-generative artificial intelligence (AI) chatbots to support students in large-enrollment undergraduate courses. We find the chatbot messaging increased students’ final grades and engagement with academic supports, such as tutoring. Treatment effects were generally consistent across student demographics, with the exception of treated women in a Microeconomics course, who earned final grades that were seven percentage points higher than wo…

The gain

In randomized trials covering 2,483 students at Georgia State University, students assigned to course chatbot texts were four percentage points more likely to earn an A or B than the 61% of control students who did.

In brief

Effects varied by course and group. In Microeconomics, women assigned to the chatbot earned final grades seven points higher than control women, were 11 percentage points more likely to earn an A or B and were six percentage points less likely to drop the course. Men in that course showed no effect. In Government, treated students were three percentage points less likely to earn a D or F or withdraw, against 18% of controls.

The chatbot did not appear to change students' wider academic habits. Researchers found no strong evidence of effects on other courses taken that term or on enrolling in another course in the subject the next term. Treated students were about two percentage points more likely to attend supplemental instruction tutoring, which explained a small share of the grade gain.

Main points

  1. The trials randomized 1,568 Government students across fall 2021, spring 2022 and fall 2022, and 915 Microeconomics students during 2022-23; all had opted in to university texting.
  2. Treated students received 2-3 customized texts a week (about 40 a semester) covering due dates, missing-assignment nudges and invitations to ask questions.
  3. Chatbot answers came from a pre-programmed knowledge base; low-confidence questions went to a teaching assistant, who also reviewed bot replies daily.
  4. Treated students were four percentage points more likely to earn an A, against 36% of controls, though after multiple-comparison correction only the gain in earning a B or higher stayed significant.
  5. In Microeconomics, treated students were three percentage points less likely to drop the course than the 8% of controls who did.
  6. Treated women in Microeconomics completed practice assignments 8-9 percentage points more often than control women; assignment completion explained 48-73% of their grade gain.
  7. Supplemental instruction attendance rose about two percentage points against baselines of 7% in Government and 2.4% in Microeconomics.
  8. On an end-of-course survey in Government, 82% of respondents across the pooled results recommended continuing the chatbot.

The full story

Georgia State University, which enrolls more than 52,000 undergraduates and where 63% of students identify as Black, Hispanic or of two or more races, chose large, asynchronous online courses with high rates of D, F or withdrawal. Half of the consenting students in each course were randomly assigned to chatbot texts, and all students still received the usual instructor emails.

The texts were personalized by whether a student had completed the prior week's work. Students could message the bot at any hour and get answers from a knowledge base built with administrators and the instructional team. The Government version added a #quizme function that offered multiple-choice practice before exams; about 38% of surveyed treated students said they used it.

The headline grade effect was similar across the two courses and across most demographic subgroups, with formal equality tests finding five significant differences in 42 comparisons. The main exception was women in Microeconomics, who ended up outperforming both control and treated men; control women and men had averaged 68.9 and 70.5.

Beyond grades, the researchers found no consistent treatment effects on attitude measures in the Government survey. Only about half of Government students answered it, and response rates were lower for Black students, course retakers and students with lower high school GPAs.

Sources

1 on file
  1. NBER Working Paperdoi:10.3386/w35397
    Peer-reviewed2026-07-01

Talk it over

Four questions to start onWho gets this benefit, and who was left out of the study or story?

Start with a question

Texts from a course chatbot raised A and B grades by 4 points in two randomized trialshttps://truvace.com/item/lets-chat-leveraging-chatbot-outreach-for-improved-course-performance

  1. Who gets this benefit, and who was left out of the study or story?

    The gain In randomized trials covering 2,483 students at Georgia State University, students assigned to course chatbot texts were four percentage points more likely to earn an A or B than the 61% of control students who did.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a randomized controlled trial with a sample of 2,483, from NBER Working Paper.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Only about half of Government students completed the end-of-course survey, with response rates lower for Black students, course retakers and students with lower high school GPAs, and no survey was fielded in Microeconomics; the authors also caution against…

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education