Truace
Subscribe

Ensemble of Domain-Specific Natural Language Processing and Large Language Models for Detecting Suicidal Ideation and Mental Health Conditions in Social Media Text: Development and Evaluation Study

Background Mental illness contributes substantially to global disability, and public adoption of AI for mental health support is accelerating without commensurate safety evaluation. General-purpose large language models can fail to recognize naturalistic text expressing suicidal ideation, with immediate clinical consequences. Domain-specific natural language processing methods offer a contrasting approach, but ensembles integrating the two have not been formally characterized. Objective We aimed to (1) benchmark…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1621· cite & verify
Ensemble of Domain-Specific Natural Language Processing and Large Language Models for Detecting Suicidal Ideation and Mental Health Conditions in Social Media Text: Development and Evaluation Study
Seattle - Sound Mental Health 01 by Joe Mabel. CC BY-SA 3.0 · http://creativecommons.org/licenses/by-sa/3.0/

In brief

Researchers compared five approaches for sorting 60,889 social media posts into nine mental health categories, including suicidal ideation. The approaches were a prompted GPT-4o-mini, a fine-tuned GPT-4o-mini, a support vector machine using symptom-informed lexical features, and two ensembles of the classifier and the fine-tuned model. On a held-out test split, the standalone classifier scored 91.9% accuracy and both ensembles scored about 93.8% to 93.9%.

The work matters because AI use for mental health support is spreading faster than safety testing, and missed suicidal ideation carries immediate clinical risk. The results suggest that clinically grounded features help stabilize language model output. The authors caution that the labels came from communities rather than clinicians and that only one dataset was used, so the findings need external validation before any use in decision support.

Main points

  1. The dataset held 60,889 Reddit and Twitter texts across nine categories, split 70%, 10% and 20% at the post level.
  2. The base prompted GPT-4o-mini reached only 58.7% accuracy, while the fine-tuned version reached 88.6%.
  3. The symptom-informed support vector machine classifier scored 91.9%, higher than both language model variants.
  4. Ensembles weighted 60% classifier and 40% fine-tuned model were selected on the validation split and reached 93.8% to 93.9% on the test split.
  5. The authors derived closed-form bounds on permissible language model weights for safety-critical ensemble weighting.

The gain

Combining a domain-specific classifier with a fine-tuned language model at a 60/40 weighting reached about 93.9% test accuracy, above either model used alone on this Reddit and Twitter corpus.

The rundown

The study used labels taken from the originating subreddit or a self-stated condition. The categories were anxiety, bipolar disorder, depression, a normal baseline, personality disorder, stress, suicidal ideation, ADHD and autism spectrum disorder.

Two fusion strategies were tested: hybrid probability-indicator fusion and soft probability fusion. Ensemble weights were chosen on the validation split, and all reported performance comes from the held-out test split. Confidence intervals were paired bootstrap intervals with 10,000 replicates, and accuracy differences were tested with exact McNemar tests.

The conclusions state that domain-specific clinical grounding remained necessary for stable ensemble performance in this corpus. Probabilistic fusion outperformed both constituent models when the language model weight was held within a derivable, model-agnostic constraint.

Sources

  1. Peer-reviewedJMIR Mental Health2026-10-08

The debate