Truace
Subscribe

Test of six AI chatbots finds 11.4% of erectile dysfunction answers safety-flagged

Study title: Large language model chatbots for patient-facing erectile dysfunction information: a cross-sectional evaluation of safety, accuracy, empathy, information quality and readability

Researchers submitted 38 standardized erectile dysfunction questions once to each of six large language model chatbot interfaces, producing 228 responses. Five independent raters scored them, and 26 responses (11.4%) were safety-flagged.

Read the study abstract

Background: Generative artificial intelligence chatbots are increasingly used for health information, including sensitive conditions, such as erectile dysfunction (ED), but their safety and quality remain uncertain. To compare six large language model chatbot interfaces for ED information across safety, accuracy, empathy, information quality, source-related transparency, readability and response length. Methods: Thirty-eight standardized ED questions were submitted once to six chatbot interfaces, yielding 228 re…

Truvace record2 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 228 · 1 independent sourcePeer-reviewedTRV-2026-2384 · cite or check it
Large language model chatbots for patient-facing erectile dysfunction information: a cross-sectional evaluation of safety, accuracy, empathy, information quality and readability"Medical Laboratory" by ben.dracup, CC BY 2.0.
How the two sides scoreContested
P 50P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

G 52G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

In brief

Accuracy was generally high, with a median of 4.00 out of 5 for every interface, though scores differed slightly overall. Empathy, DISCERN, Ensuring Quality Information for Patients and Journal of the American Medical Association benchmark scores all differed significantly between chatbots, while Global Quality Score did not.

The authors conclude the chatbots may support information seeking but should not replace individualized clinical assessment. Safety-flag frequency differed across interfaces, but the effect size was small and no pairwise difference held after multiplicity correction.

Main points

  1. Thirty-eight standardized questions were each submitted once to six chatbot interfaces, yielding 228 responses.
  2. Twenty-six responses (11.4%) were safety-flagged by the raters.
  3. Safety-flag frequency differed across interfaces (P = 0.038) with a small effect size of 0.062, and no pairwise difference remained significant after correction.
  4. Accuracy had a median of 4.00 out of 5 for all interfaces, with a slight overall difference (P = 0.010, Kendall's W = 0.080).
  5. The JAMA benchmark showed the largest spread between chatbots (W = 0.888), followed by Ensuring Quality Information for Patients (W = 0.576) and DISCERN (W = 0.508).
  6. Global Quality Score did not differ significantly across interfaces (P = 0.146).
  7. Readability and response length differed between interfaces (all P < 0.001).

The gain

In a test of six chatbot interfaces answering 38 erectile dysfunction questions, accuracy was generally high, with a median of 4.00 out of 5 for every interface.

The problem

In a test of six chatbot interfaces answering 38 erectile dysfunction questions, raters safety-flagged 26 of 228 responses (11.4%), and empathy, information quality and readability differed across the chatbots.

The full story

The study was a cross-sectional evaluation of six large language model chatbot interfaces answering patient-facing questions on erectile dysfunction, a sensitive condition for which people increasingly turn to generative AI. Each of 38 standardized questions was submitted once, giving 228 responses.

Five independent raters scored responses on Safety, Accuracy and Empathy, plus four information-quality tools: DISCERN, Ensuring Quality Information for Patients, the JAMA benchmark and Global Quality Score. Readability and response length were also measured, and differences between interfaces were tested with Cochran's Q or Friedman tests.

Results showed 11.4% of responses were safety-flagged. Differences in flag frequency across interfaces were statistically detectable (Q = 11.765, P = 0.038) but small, and none survived multiplicity correction in pairwise comparisons. Accuracy stayed high across interfaces, while empathy, source-related transparency and information quality scores varied more.

The authors conclude the chatbots generally gave accurate information and may help people seek it, but performance varied on clinically relevant dimensions and the tools should not replace individualized clinical assessment.

What this doesn’t fix

Each question was submitted only once to each chatbot interface, so the results capture a single response per question rather than repeated or varied answers.

Sources

  1. Peer-reviewedSexual Health2026-12-17

The debate