aceModerate evidence
More
25 experiments show social skills lift health chatbot ratings and psychological outcomes
Study title: Effectiveness of Communication Competence in AI Conversational Agents for Health: Systematic Review and Meta-Analysis
- Evidence type
- Systematic review or meta-analysis
- Sample
- n = 4,525
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2737cite or check it
Study title: Effectiveness of Communication Competence in AI Conversational Agents for Health: Systematic Review and Meta-Analysis
A systematic review and meta-analysis in the Journal of Medical Internet Research pooled 25 experiments with 4,525 participants. They compared health chatbots with more or less communication competence, such as empathetic replies, small talk and self-disclosure. Chatbots with more of these skills were rated better (Hedges g=0.45) and produced better psychological outcomes (g=0.49).
Read the study abstract
Background: With advancements in artificial intelligence and large language models, researchers and designers have increasingly focused on enhancing the conversational capacity of health-related conversational agents (CAs). Communication competence, a key concept in interpersonal communication influencing relational and health outcomes, has been extended to human-machine communication to emphasize the CAs' ability to demonstrate appropriate communicative behaviors in managing relationships with humans. Objective…
The gain
A meta-analysis of 25 experiments with 4,525 participants found that health chatbots built with more communication skills, such as empathy, were rated higher (Hedges g=0.45) and led to better psychological outcomes (g=0.49) than less skilled versions.
In brief
The searches covered 7 databases and 8,309 papers up to the end of 2023 and yielded 31 experimental studies. The authors identified 11 strategies for building these skills; empathetic response was the most common, tested in 18 studies. Most studies were in mental health (17 of 31) and most used text-based chatbots.
The benefits did not extend to behaviour or health. Communication skills had no significant effect on how much people used the chatbots (g=0.11) or on health outcomes (g=0.18). Health outcomes came from only 3 studies, so that result is preliminary.
Main points
- The review pooled 25 experiments with 4,525 participants (mean age 29.7), drawn from 31 studies found in 8,309 papers across 7 databases.
- User evaluations of chatbots improved with more communication skills (Hedges g=0.45, 95% CI 0.24-0.66), based on 105 effect sizes.
- Psychological outcomes, including emotions, emotional validation and stress reduction, improved (g=0.49, 95% CI 0.19-0.78).
- Use of the chatbots (g=0.11, 95% CI -0.05 to 0.26) and health outcomes (g=0.18, 95% CI -0.13 to 0.50) showed no significant effect.
- Empathetic response was the most tested of 11 strategies (18 studies), followed by self-disclosure (6), contingency (5), small talk (5), social etiquette (4) and explanation (4).
- Effects were stable across participants' average age, type of interaction and health topic; later publication year was linked to smaller effects on use and psychological outcomes.
- Risk of bias was low in 1 study, raised some concerns in 13 and was high in 11 of the 25 meta-analysed studies.
The full story
The authors included experiments that manipulated a chatbot's communication competence in health conversations, recruited human participants and reported at least one relevant outcome. Chatbots with avatars, robots or click-only menus were excluded. Of the 31 studies, 21 used between-subjects designs, 24 had participants use a chatbot built for the study, 4 used a human-operated 'Wizard of Oz' setup and 3 used prerecorded conversations. Participants came from countries including the United States, China, Germany, Belgium and the Netherlands.
Effect sizes were pooled with three-level random-effects models. The two significant findings were for user evaluations (g=0.45) and psychological outcomes (g=0.49). Heterogeneity was significant for all four outcomes. Moderator analyses found no effect of age, interaction type or health topic on evaluations. Publication year was a significant moderator for use of the chatbots (b=-0.04) and for psychological outcomes (b=-0.26), with smaller effects in more recent studies.
Health outcomes were mostly intentions, such as intending to cook recommended recipes or get vaccinated, and openness to seeking medical help. All 21 health-outcome effect sizes came from 3 studies. The pooled effect ranged from 0.00 to 0.44 when single studies were removed, so no moderator analysis was run for it. The authors call for studies that measure real behaviour change, treatment adherence and symptom improvement.
Sources
1 on file- Journal of Medical Internet Researchdoi:10.2196/76296
Talk it over
Who gets this benefit, and who was left out of the study or story?
25 experiments show social skills lift health chatbot ratings and psychological outcomeshttps://truvace.com/item/effectiveness-of-communication-competence-in-ai-conversational-agents-for-health
Who gets this benefit, and who was left out of the study or story?
The gain A meta-analysis of 25 experiments with 4,525 participants found that health chatbots built with more communication skills, such as empathy, were rated higher (Hedges g=0.45) and led to better psychological outcomes (g=0.49) than less skilled versions.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a systematic review or meta-analysis with a sample of 4,525, from Journal of Medical Internet Research.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat Health outcomes were analysed in only 3 of the 25 studies and the pooled estimate shifted from 0.00 to 0.44 when single studies were removed, so the lack of a health effect is preliminary.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say