Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-1702
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-17T10:43:02.529456Z
status: published
lens: trace
sector: health
headline: Large Language Model Chatbot Responses to Cancer Survivorship Questions in Hong Kong: Bilingual Evaluation and Prompt Optimization Study
dek: Background As the population of survivors of cancer grows and new information technologies become widespread in Hong Kong, survivors of cancer increasingly seek ongoing care and support information from large language models (LLMs). While these tools provide immediate conversational responses, they carry substantial risks of generating inaccurate, unsafe, or generic medical advice. Evaluating LLM response performance is particularly critical in Hong Kong's bilingual health care context. Objective This study aime…
gain_title: In an exploratory within-sample comparison, an expert-designed CRAFT prompt raised Gemini 3.1 Pro's clinical utility for cancer survivorship questions and corrected baseline unsafe outputs without reducing understandability.
problem_title: Unconstrained chatbots gave unsafe cancer survivorship answers in both languages, with up to 24% English errors and up to 20% Traditional Chinese errors.
trace_subject: LLM chatbot answers to cancer survivorship questions in English and Traditional Chinese
gain_reading: In an exploratory within-sample comparison, an expert-designed CRAFT prompt raised Gemini 3.1 Pro's clinical utility for cancer survivorship questions and corrected baseline unsafe outputs without reducing understandability.
gain_evidence: expert-designed CRAFT prompting improved clinical utility and corrected the unsafe outputs seen at baseline without reducing understandability | significantly improved clinical utility across both languages
problem_reading: Unconstrained chatbots gave unsafe cancer survivorship answers in both languages, with up to 24% English errors and up to 20% Traditional Chinese errors.
problem_evidence: English errors (up to 24%) centered on service mismatches and definitive interpretations | Traditional Chinese errors (up to 20%) involved prescriptive disease management and unverified adjunctive therapies
quick_read: Researchers tested four commercial language models on 25 expert-curated cancer survivorship questions spanning five domains, in both English and Traditional Chinese. A Delphi expert panel scored the baseline answers for clinical utility, safety, understandability and readability. Google Gemini 3.1 Pro was the most clinically useful in both languages.

The findings matter because cancer survivors in Hong Kong increasingly turn to chatbots for ongoing care information, and the baseline models made different kinds of unsafe errors in each language. A structured prompt designed by experts appeared to reduce those errors. The prompt result is exploratory and was tested only within the same sample of questions, so wider reliability is not established.
limitation: The prompt comparison was an exploratory within-sample analysis, and the link between readability and clinical utility was not formally tested, so the results should not be treated as confirmed general performance.
tag: Dual reading
key_points: Four models were tested: Google Gemini 3.1 Pro, HK Chat-0.6.2, DeepSeek-V3.2 and Kimi-K2.5. | Gemini 3.1 Pro scored a mean of 3.99/5 in English and 4.21/5 in Traditional Chinese for clinical utility at baseline. | English errors involved service mismatches and definitive interpretations; Traditional Chinese errors involved prescriptive disease management and unverified adjunctive therapies. | The most readable models in each language were not the ones with the highest clinical utility, though this was not formally tested. | With the CRAFT prompt, Gemini 3.1 Pro's English clinical utility mean rose to 4.68 (SD 0.18).
rundown: The study used two phases. Phase 1 fed 25 questions to four models and had experts rate clinical utility on a 5-point scale and safety as a binary category. Understandability was measured with the PEMAT, and readability with Flesch-Kincaid Grade Level for English and lexical richness for Traditional Chinese.

Phase 2 had experts build a CRAFT prompt covering context, role, audience, format, task and tone. The responses were compared with baseline outputs using paired statistical tests in an exploratory analysis. The authors report that the prompt improved clinical utility in both languages and fixed the unsafe outputs seen at baseline.

The authors conclude that safe use of LLMs in multilingual oncology settings requires strict validation along with structured, role-restrictive constraints.
sources:
- peer_reviewed | Journal of Medical Internet Research | https://doi.org/10.2196/105778 | 2026-10-08
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

4adf35795b3aa69e3b4234e23b1e73d031f6101c01055fe0e645f8d7b19e62e0

previous

0000000000000000000000000000000000000000000000000000000000000000