Truace
Subscribe

AI chatbot answers on diabetes remission trade off readability against accuracy

Study title: Quality and readability of large language model–generated responses on type 2 diabetes remission: a cross-sectional balanced factorial study

Researchers asked four AI chatbots, DeepSeek-V3.2, GPT-5.5, Gemini 3.1 Pro and Claude Sonnet 4.6, 16 questions about type 2 diabetes remission, using six prompting styles. That produced 384 responses. Two endocrinologists rated each on accuracy, relevance, comprehensiveness, empathy and avoiding falsehoods, and readability was scored with standard formulas.

Read the study abstract

Abstract Large language models (LLMs) are increasingly being explored for patient education and medical information delivery. However, their outputs may involve trade-offs among medical accuracy, comprehensiveness, empathy, and readability. Type 2 diabetes remission is a clinically nuanced concept that is readily misunderstood by patients, making it a suitable context in which to evaluate both the medical quality and patient accessibility of artificial intelligence–generated health information. To systematically…

By the Truvace desk1 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 384 · 1 independent sourcePeer-reviewedTRV-2026-2315· cite & verify
Quality and readability of large language model–generated responses on type 2 diabetes remission: a cross-sectional balanced factorial study
FreeStyle libre am Oberarm und Auslesegerät-3 by Thirunavukkarasye-Raveendran. CC BY 4.0 · https://creativecommons.org/licenses/by/4.0

In brief

The findings matter because patients may rely on chatbots for health information, and the study shows that the choice of model and of prompt changes what they receive. Gains in readability or empathy came with losses elsewhere. The results cover one narrow question set, and the correlation analysis was exploratory, so they cannot be taken as general proof.

Main points

  1. Composite quality scores ranged from 4.175 for Gemini 3.1 Pro to 4.285 for DeepSeek-V3.2.
  2. GPT-5.5 had the highest mean accuracy (4.448) and falsehood avoidance (4.469).
  3. The 6th-grade prompt gave the easiest text (FRES 69.70), while the Doctor prompt gave the hardest (27.50).
  4. Prompting strategy had its largest effect on empathy (LR χ² = 243.836).
  5. Readability correlated positively with empathy (ρ = 0.53) but negatively with comprehensiveness (ρ = −0.46).

The problem

Chatbot answers about type 2 diabetes remission could not be optimized on every measure at once, and the easiest-to-read answers scored lower on comprehensiveness and falsehood avoidance.

The rundown

The study used a balanced factorial design: four models crossed with six prompting strategies (Keyword, Question, 6th-grade, Guideline, Patient and Doctor) across 16 core questions. All 384 responses were analyzed with no missing data.

Two endocrinologists with more than 5 years of experience independently rated responses on 5-point scales using a prespecified rubric. Quality ratings were analyzed with cumulative link mixed models and readability with linear mixed-effects models, with question as a random intercept.

DeepSeek-V3.2 scored highest on relevance (4.948), comprehensiveness (4.427) and empathy (3.615). Claude Sonnet 4.6 produced the hardest text on average (FRES 32.98), compared with 51.28 for DeepSeek-V3.2. The Patient prompt gave the highest empathy (4.359) and the Doctor prompt the highest comprehensiveness (4.609).

Sources

  1. Peer-reviewedScientific Reports2026-10-29

The debate