LLM chatbot answers to cancer survivorship questions in English and Traditional Chinese

Source article: Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study

Background As the population of survivors of cancer grows and new information technologies become widespread in Hong Kong, survivors of cancer increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual health care context. Objective This study aime…

By the Truvace deskOctober 17, 20262 min read1 sourcePeer-reviewedTRV-2026-1702· cite & verify

Trace impact readingContested

P 68

Both sides are scored from claims and sources, not community votes.

G 71

In brief

Researchers tested four commercial language models on 25 expert-curated cancer survivorship questions spanning five domains, in both English and Traditional Chinese. A Delphi expert panel scored the baseline answers for clinical utility, safety, understandability and readability. Google Gemini 3.1 Pro was the most clinically useful in both languages.

The findings matter because cancer survivors in Hong Kong increasingly turn to chatbots for ongoing care information, and the baseline models made different kinds of unsafe errors in each language. A structured prompt designed by experts appeared to reduce those errors. The prompt result is exploratory and was tested only within the same sample of questions, so wider reliability is not established.

Main points

Four models were tested: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2 and Kimi-K2.5.
Gemini 3.1 Pro scored a mean of 3.99/5 in English and 4.21/5 in Traditional Chinese for clinical utility at baseline.
English errors involved service mismatches and definitive interpretations; Traditional Chinese errors involved prescriptive disease management and unverified adjunctive therapies.
The most readable models in each language were not the ones with the highest clinical utility, though this was not formally tested.
With the CRAFT prompt, Gemini 3.1 Pro's English clinical utility mean rose to 4.68 (SD 0.18).

The gain

In an exploratory within-sample comparison, an expert-designed CRAFT prompt raised Gemini 3.1 Pro's clinical utility for cancer survivorship questions and corrected baseline unsafe outputs without reducing understandability.

The problem

Unconstrained chatbots gave unsafe cancer survivorship answers in both languages, with up to 24% English errors and up to 20% Traditional Chinese errors.

The rundown

The study used two phases. Phase 1 fed 25 questions to four models and had experts rate clinical utility on a 5-point scale and safety as a binary category. Understandability was measured with the PEMAT, and readability with Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese.

Phase 2 had experts build a CRAFT prompt covering context, role, audience, format, task and tone. The responses were compared with baseline outputs using paired statistical tests in an exploratory analysis. The authors report that the prompt improved clinical utility in both languages and fixed the unsafe outputs seen at baseline.

The authors conclude that safe use of LLMs in multilingual oncology settings requires strict validation along with structured, role-restrictive constraints.

What this doesn’t fix

The prompt comparison was an exploratory within-sample analysis, and the link between readability and clinical utility was not formally tested, so the results should not be treated as confirmed general performance.