Truace
Subscribe

Cervical cancer chatbot made fewer factual errors when grounded in guidelines, a study finds

Study title: A Multilingual AI-Driven Conversational System for Cervical Cancer Awareness and Risk Assessment: Development and Evaluation Study

Researchers built a three-part digital health platform for cervical cancer awareness: a rule-based risk questionnaire, a chatbot powered by GPT-4o mini that retrieves guideline material before answering, and a module that recommends nearby health care facilities. In clinician review of 50 responses, the retrieval-grounded chatbot had factual errors in 8% of outputs, against 28% for a baseline without retrieval.

Read the study abstract

Background: Cervical cancer is the fourth most common cancer among women globally, with an estimated 660,000 new cases and 350,000 deaths in 2022. It imposes a disproportionate burden on underserved populations in low- and middle-income countries, where language barriers, cultural stigma, and limited access to health care hinder awareness and uptake of preventive measures. Objective: This study aimed to design, develop, and evaluate a multilingual AI-driven dialogue system for cervical cancer awareness, personal…

Truvace record2 min read1 sourceEvidence: Emerging · 48/100 — Benchmark or lab evaluation · n = 50 · 1 independent sourcePeer-reviewedTRV-2026-2420 · cite or check it
A Multilingual AI-Driven Conversational System for Cervical Cancer Awareness and Risk Assessment: Development and Evaluation Study"Medical Laboratory" by ben.dracup, CC BY 2.0.

In brief

The work targets underserved populations in low- and middle-income countries, where language barriers, cultural stigma and limited access to care hinder awareness and preventive uptake. Cervical cancer is the fourth most common cancer among women globally, with an estimated 660,000 new cases and 350,000 deaths in 2022. The system supports English, Spanish, Yoruba and Swahili; the first three were included in the multilingual evaluation.

The evaluation measured technical performance only. The authors state that effectiveness for awareness, risk literacy and screening access remains to be evaluated in community settings, and they plan large-scale community validation, bias auditing and integration with public health screening programs.

Main points

  1. The chatbot's retrieval precision@3 was 0.88, and its hallucination rate fell 71.4% compared with a non-RAG baseline.
  2. Independent clinicians reviewing 50 responses found factual errors in 4 RAG-grounded answers (8%) versus 14 (28%) for the baseline, with Cohen κ of 0.84.
  3. An automated LLM-as-judge rated factual correctness at 4.3 out of 5.
  4. The risk engine uses a 30-item questionnaire mirroring the UCI Cervical Cancer Risk Factors dataset, with weights drawn from WHO, CDC and American Cancer Society guidance.
  5. The engine gave low, moderate or high risk levels with per-factor explanations for every completed assessment.
  6. The geospatial facility recommendation module reached a specialty relevance precision of 0.87.
  7. Backend API latency averaged 312 ms (SD 47) under a simulated load of 50 concurrent users.

The gain

In a development study, clinicians reviewing 50 responses found factual errors in 8% (4/50) of a guideline-grounded cervical cancer chatbot's answers versus 28% (14/50) for a non-grounded baseline.

The full story

The team used a design science research framework to build a three-tier, cross-platform system in TypeScript (NestJS) and Flutter, deployed on Android, iOS and web. Its risk engine is rule-based and transparent, so each risk level can be traced to specific factors rather than a black-box prediction.

The chatbot was tested with retrieval precision, hallucination rate, LLM-as-a-judge scoring, adversarial safety testing and native-speaker ratings of multilingual output. The risk engine was assessed through clinical face-validity review of its factor weights rather than against patient outcomes.

Results showed the retrieval-grounded chatbot made fewer errors than a non-RAG baseline, with 4 of 50 clinician-reviewed answers containing errors against 14 of 50. The geospatial module reached 0.87 specialty relevance precision, and the backend handled a simulated 50 concurrent users at 312 ms average latency.

Sources

  1. Peer-reviewedJMIR Cancer2026-11-06

The debate