AI chatbots and depressive symptoms: a living meta-analysis of randomized trials

1 Question and eligibility

In randomized trials, do people who are given a mental-health chatbot end up with fewer depressive symptoms than people who are not?

A trial is included when

• Randomized allocation to a chatbot group and to a comparison group that fits the comparator above.
• The chatbot programme targets depression, anxiety, stress or psychological wellbeing.
• Depressive symptoms reported on a validated scale after the programme.
• Published in a peer-reviewed journal, in English, from 1 January 2016 on.
• The numbers needed (below) can be read in the abstract or in an open-access full text.

A trial is left out when

• Not randomized: single-group studies, before-and-after studies and quasi-experiments.
• Every comparison group is another chatbot, another digital programme or human-delivered therapy.
• The chatbot is a small part of a larger programme (for example an app with human coaching), or people secretly write its replies.
• The programme targets something else (smoking, weight, medication, physical activity, education), even if mood was measured.
• Protocols, conference abstracts, preprints, secondary analyses of a trial already included and retracted papers.
• No usable numbers in the abstract or in an open-access full text.

2 Information sources and search

Publications dated 1 January 2016 up to the day of the search.

PubMed

(chatbot*[tiab] OR "chat bot*"[tiab] OR "conversational agent*"[tiab] OR "conversational AI"[tiab] OR "dialogue system*"[tiab] OR "dialog system*"[tiab] OR "relational agent*"[tiab] OR "virtual agent*"[tiab] OR "large language model*"[tiab] OR "generative AI"[tiab] OR "generative artificial intelligence"[tiab] OR ChatGPT[tiab] OR Woebot[tiab] OR Wysa[tiab] OR Therabot[tiab]) AND (depress*[tiab] OR "mental health"[tiab] OR "psychological distress"[tiab] OR anxiety[tiab] OR wellbeing[tiab] OR "well-being"[tiab]) AND (randomized controlled trial[pt] OR randomized[tiab] OR randomised[tiab] OR "randomly assigned"[tiab] OR RCT[tiab]) AND ("2016/01/01"[dp] : "3000"[dp])

Europe PMC

(TITLE:(chatbot* OR "chat bot" OR "conversational agent" OR "conversational agents" OR "conversational AI" OR "relational agent" OR "large language model" OR "generative AI" OR ChatGPT OR Woebot OR Wysa OR Therabot) OR ABSTRACT:(chatbot* OR "chat bot" OR "conversational agent" OR "conversational agents" OR "conversational AI" OR "relational agent" OR "large language model" OR "generative AI" OR ChatGPT OR Woebot OR Wysa OR Therabot)) AND (TITLE:depress* OR ABSTRACT:depress*) AND (TITLE:(randomized OR randomised OR "randomly assigned" OR RCT) OR ABSTRACT:(randomized OR randomised OR "randomly assigned" OR RCT)) AND PUB_YEAR:[2016 TO 2030] AND NOT SRC:PPRPreprints are left out of the search itself.

OpenAlex

(chatbot OR "conversational agent" OR "conversational AI" OR Woebot OR Wysa OR Therabot) AND (depression OR depressive) AND ("randomized controlled trial" OR "randomised controlled trial")Title and abstract search, journal articles from 2016 on.

Truvace's own records

(chatbot OR conversational agent) AND depress AND randomPublished records whose headline, abstract or readings match, so nothing the site already holds is missed.

3 Selection and data extraction

• Results from all sources are merged and duplicates removed by PubMed ID, DOI and title.
• Stage 1, titles and abstracts: an AI model applies the inclusion and exclusion rules above to each result and records a decision with its reason. Anything unclear goes forward to stage 2.
• Stage 2, eligibility and numbers: for each result that went forward, the abstract and any open-access full text are read against the same rules, and the numbers are extracted with their quotes. The code then checks every quote against the stored text.
• Every result is counted in the flow diagram, and every exclusion at stage 2 is listed with its reason.
• Before an analysis is published, a person checks each included trial's numbers against the quotes shown beside them.

4 Outcome and data rules

Which result is used

• One effect per trial.
• Scale: the depression scale the paper names as primary. If it names none, the first available of PHQ-9 or PHQ-8, BDI or BDI-II, CES-D, DASS-21 depression, HADS depression, then any other validated depression scale.
• Time point: the first assessment after the programme ends. Later follow-ups are not used.
• Several chatbot groups: the group the paper presents as the full or main chatbot; if it names none, the first chatbot group listed.
• Several comparison groups: an information-only group is preferred over a waiting list or no intervention, because it is the harder test.
• Direction: negative values mean fewer depressive symptoms in the chatbot group. Scales where higher is better are reversed.

Which numbers are used

• Preferred: each group's mean and standard deviation after the programme, with the number analysed (intention-to-treat figures when the paper gives both).
• If a group's spread is given as a standard error or a 95% confidence interval of the mean, it is converted to a standard deviation (Cochrane Handbook 6.5.2).
• If only change from baseline is given (mean and standard deviation per group), change scores are used and the trial is marked.
• If none of those are available but the paper reports a standardized mean difference between groups with a 95% confidence interval, that is used and the trial is marked.
• Nothing is imputed, read off a figure, back-calculated from a p value or requested from authors.
• Every number is stored with the sentence or table row it came from, word for word. A number whose quote the code cannot find in the stored source text is not used.
• The effect size is Hedges g: the difference in means over the pooled standard deviation, with the small-sample correction.

5 Analysis plan

• Random-effects model. Between-trial variance (tau²) by restricted maximum likelihood.
• 95% confidence interval for the pooled effect with the Hartung-Knapp adjustment.
• Heterogeneity reported as tau², I² and Cochran's Q.
• A 95% prediction interval (Higgins, Thompson and Spiegelhalter) when there are at least three trials.
• Leave-one-out: the analysis is repeated with each trial removed in turn.
• With ten or more trials, Egger's regression test for small-study effects. With fewer, no such test is run.
• The analysis is published only with at least five trials.

Sensitivity analyses and subgroups

• DerSimonian-Laird variance with a normal-theory interval, to show how much the result depends on the model.
• Only trials with post-programme means and standard deviations (dropping change-score, reported-effect and cluster-randomized trials).
• By comparison group (information-only against waiting list, no intervention or usual care) and by chatbot type (generative against scripted or retrieval-based), each shown only when both groups hold at least three trials.

6 Keeping it current

• The same searches are re-run about once a month. New results go through the same two stages.
• A new eligible trial produces a new draft of the analysis. Nothing changes on the public page until an editor approves the draft.
• Every approved version stays public with its date, its trials and its result.
• If the rules themselves have to change, the protocol gets a new version number and the reason is stated on the page.

7 Known limits

• Embase, PsycINFO and CENTRAL need a subscription and were not searched, so some trials will be missing.
• Trials whose numbers sit only in a paywalled full text are left out. That can skew the result if paywalled trials differ from open ones.
• No formal risk-of-bias rating (such as RoB 2) is done in this version. Trials differ in dropout, blinding and how outcomes were analysed, and the pooled figure does not adjust for that.
• Screening and extraction are done by an AI model against these rules, with code checking every quote and a person checking the numbers before publication. That is not the same as two independent human reviewers.
• Depressive symptoms are self-reported on different scales in different populations, and comparison groups range from waiting lists to e-books. A standardized mean difference puts them on one scale but does not make the trials the same.
• The person who wrote this protocol had already read some of these trials and earlier reviews of them.