Truace
Subscribe

Psychiatrists compared small Korean counseling AI models with GPT-4o on 50 scenarios

Study title: Optimizing Small Local Language Models for Culturally Competent Mental Health Counseling: Comparative Evaluation With GPT-4o by Psychiatrists

Researchers fine-tuned three Korean-optimized models of roughly 7 to 8 billion parameters on 1558 diary-based counseling interactions from a Korean mental health app. Ten board-certified psychiatrists then rated the tuned models and GPT-4o on 50 counseling scenarios across four dimensions.

Read the study abstract

Background: Large language models (LLMs) show promise for mental health counseling applications. However, their deployment faces significant challenges, including data privacy concerns requiring on-premise solutions, high operational costs, and Western-centric biases that limit cultural appropriateness for non-English-speaking populations. Objective: This study aimed to develop and evaluate culturally competent small language models (SLMs) with approximately 7 to 8 billion parameters for Korean mental health cou…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-2247· cite & verify
Peer-reviewed study: JMIR AI
Trace impact readingNegative state
P 71The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 65The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

The best local model was statistically indistinguishable from GPT-4o on supportiveness and cultural appropriateness, which suggests that privacy-preserving on-premise counseling tools may be feasible for non-English-speaking populations. The safety shortfall, the small scenario set, and reliance on good language-specific base models leave open how far these tools can be used without clinician supervision.

Main points

  1. Three base models (SKT A.X 4.0 Light, LG EXAONE 3.5, Meta Llama 3.1) were fine-tuned with high-rank LoRA (r=256) over 20 epochs.
  2. SKT A.X 4.0 Light scored mean 3.30 versus 3.29 for GPT-4o in cultural appropriateness (P=.58).
  3. Psychiatrist agreement was higher for the local model (ICC=0.575) than for GPT-4o (ICC=0.211) on cultural appropriateness.
  4. GPT-4o had significantly higher safety scores than every small model (P<.001).
  5. The GPT-5 LLM-as-a-Judge correlated moderately to well with humans but showed a ceiling effect in safety scoring.

The gain

A fine-tuned 7B Korean small language model showed no statistically significant difference from GPT-4o in supportiveness and cultural appropriateness, as rated by psychiatrists, supporting local, privacy-preserving deployment.

The problem

All tested small language models scored significantly lower than GPT-4o on safety in mental health counseling scenarios, so human expert oversight is still needed.

The rundown

The dataset came from the Korean mental health mobile app Maeumpyeonuijeom. Inference parameters were tuned with GPT-5 acting as an LLM-as-a-Judge, a screening step that preceded expert clinical validation in a 2-stage hybrid evaluation framework.

The evaluation used 5-point Likert scales for individualization, supportiveness, cultural appropriateness, and safety. Statistical comparison relied on Wilcoxon signed-rank tests with Holm correction, and interrater reliability was assessed with intraclass correlation coefficients.

The study motivates local models by citing data privacy concerns that require on-premise solutions, high operational costs, and Western-centric biases that limit cultural appropriateness. The authors conclude that success depends on the availability of high-quality language-specific base models.

What this doesn’t fix

The safety gap versus GPT-4o means the small models need human expert oversight, and the LLM-as-a-Judge showed a ceiling effect in safety scoring, limiting automated screening for safety.

Sources

  1. Peer-reviewedJMIR AI2026-10-22

The debate