Truace
Subscribe

14 practitioners helped refine an AI depression tool, yet flagged bias and missed cues

Study title: A Large Language Model-Based System for Depression Symptom Identification: Iterative Qualitative Co-Design Studies

Researchers built a depression symptom identification model on a fine-tuned GPT-3.5 and refined it over two rounds of online workshops with mental health practitioners. The first version covered 5 symptom categories and was reviewed by 6 practitioners. A second version with 10 categories was reviewed by 8 more.

Read the study abstract

Background: AI technologies have considerable potential to support mental health practitioners in identifying depression symptoms. However, existing tools are often developed with limited practitioner involvement and evaluated primarily on technical performance, leaving questions of clinical relevance, ethical design, and practitioner trust insufficiently addressed. Objective: This study aimed to examine how iterative co-design with mental health practitioners can inform the development of an AI-based model for…

By the Truvace desk1 min read1 sourceEvidence: Emerging · 37/100 — Interviews or qualitative study · n = 14 · 1 independent sourcePeer-reviewedTRV-2026-2361· cite & verify
A Large Language Model-Based System for Depression Symptom Identification: Iterative Qualitative Co-Design Studies
Seattle - Sound Mental Health 02 by Joe Mabel. CC BY-SA 3.0 · http://creativecommons.org/licenses/by-sa/3.0/

In brief

The work suggests that involving clinicians early can make such tools more clinically relevant, but it measures practitioner perceptions rather than diagnostic performance. Risk categorization, missing nonverbal cues, cultural differences and automation bias remain unresolved, and the authors call for independent validation and safety evaluation.

Main points

  1. Stage 1 used DepressionSymp 1.0, a fine-tuned GPT-3.5 trained on the CounselChat dataset, with 5 symptom categories.
  2. Practitioners saw indirect indicators such as social withdrawal and financial difficulties as lifestyle factors rather than symptoms, which led to expansion to 10 categories.
  3. Risk categorization was the most ethically sensitive feature across both stages.
  4. Practitioners raised concerns about missing nonverbal information, cultural and linguistic differences, and automation bias.

The gain

Eight mental health practitioners said the expanded 10-category version of the tool was clearer and more useful for preassessment than the first version.

The rundown

The study used a two-stage qualitative co-design approach. In stage 1, 6 practitioners discussed the first model in online workshops. The categories were based on DSM-5 criteria and the LIWC-22 psycholinguistic framework.

Feedback from stage 1 shaped DepressionSymp 2.0. It expanded from 5 to 10 symptom categories and was reviewed in a second round by 8 further practitioners, who found it clearer and more useful for preassessment.

The authors draw design implications for LLM features, practitioner-in-the-loop oversight, and ethical issues including clinical standards, informed consent, and data governance. They recommend larger-scale independent validation, formal safety evaluation, and culturally diverse datasets.

Sources

  1. Peer-reviewedJMIR Formative Research2026-11-03

The debate