Truace
Subscribe

Information quality, understandability, actionability and readability of large language models for HIV health education: A cross-platform evaluation

Objectives: Large language models (LLMs) are increasingly used to generate health education materials, but their suitability for patient-facing human immunodeficiency virus (HIV) education remains insufficiently evaluated. We assessed the information quality, understandability, actionability, readability, text structure and source transparency of HIV patient education materials generated by six LLMs or artificial intelligence platforms. Methods: In this cross-sectional study, 100 common HIV patient education que…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1874· cite & verify
Information quality, understandability, actionability and readability of large language models for HIV health education: A cross-platform evaluation
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.

In brief

Researchers submitted 100 common HIV patient questions across 10 domains to six AI platforms, producing 600 English responses. They scored the outputs on understandability, actionability, information quality, readability, text structure and source transparency using several established tools.

The results suggest these tools should be treated as drafting aids that clinicians review rather than stand-alone sources for patients. Because factual accuracy, guideline concordance, hallucination and clinical safety were not assessed, the study cannot say whether the content was correct or safe.

Main points

  1. The study covered six platforms: ChatGPT 5.4-Thinking, DeepSeek-V3.2, Doubao, ERNIE Bot 5.0 Preview, Ant Afu and Gemini 3.1 Pro.
  2. Platform differences were significant after family adjustment for all prespecified text-structure, quality/understandability/actionability and readability outcomes.
  3. At topic level, PEMAT-P understandability and actionability and all seven readability indices differed significantly, while EQ IP-36, DISCERN, GQS, AIRM and JAMA scores did not.
  4. Overall actionability was limited and readability demands were high.

The problem

Chatbot-generated HIV patient education varied widely across platforms and topics, with limited actionability and high readability demands, so it is not suited to serve as an independent patient information source.

The rundown

The cross-sectional design asked each of 100 questions once in English, giving 600 responses in total. Evaluation used PEMAT-P, EQ IP-36, DISCERN, Global Quality Score, an AI response quality metric, JAMA benchmarks and seven readability indices.

Statistical comparisons between platforms used paired Friedman tests with Wilcoxon-Holm post hoc analyses. Topic analyses aggregated at question level and used Kruskal-Wallis and Dunn-Holm comparisons.

The authors conclude that substantial heterogeneity across platforms and topics supports using LLMs as expert-reviewed drafting tools rather than independent patient education sources.

Sources

  1. Peer-reviewedHIV Medicine2026-10-18

The debate