Emerging evidence
More
Test of six AI chatbots finds 4 of 54 epididymitis answers had potentially harmful advice
Study title: Quality, readability, and potential safety risks of epididymitis information generated by six artificial intelligence chatbots: a cross-sectional comparative study
- Evidence type
- Benchmark or lab evaluation
- Sample
- n = 54
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2932cite or check it
Study title: Quality, readability, and potential safety risks of epididymitis information generated by six artificial intelligence chatbots: a cross-sectional comparative study
Researchers submitted nine popular epididymitis search queries, drawn from 25 Google Trends queries, once each to ChatGPT, Gemini, Perplexity, Copilot, Kimi and DeepSeek. Of the 54 responses, 4 (7.4%) contained potentially harmful information, all about antimicrobial recommendations.
Read the study abstract
Background AI chatbots are increasingly used for medical information, but the quality, readability, and safety of disease-specific responses remain uncertain. We compared epididymitis information generated by six widely accessible AI chatbots. Methods Nine of 25 Google Trends queries identified worldwide from August 2021 to August 2026 were each submitted once to ChatGPT, Gemini, Perplexity, Copilot, Kimi, and DeepSeek. Responses were assessed with DISCERN, EQIP, JAMA, GQS, and six readability indices. Between-m…
The problem
In a cross-sectional test of six AI chatbots answering nine epididymitis queries (54 responses), 4 responses (7.4%) contained potentially harmful information, all on antimicrobial recommendations, and none met sixth-grade readability benchmarks.
In brief
Transparency was poor: JAMA benchmark scores were 0 in 52 of 54 responses (96.3%), meaning the answers did not meet the scale's criteria for disclosing sources and authorship. None of the responses met sixth-grade reading benchmarks, so the material may be hard for many patients to understand.
The chatbots differed on quality scores, with Copilot posting the highest median EQIP score (70.00), but no pairwise comparison between models remained significant after correction. The authors call the work an exploratory snapshot and say chatbots should not replace professional assessment or guideline-based care.
Main points
- Six chatbots were tested: ChatGPT, Gemini, Perplexity, Copilot, Kimi and DeepSeek.
- Nine of 25 worldwide Google Trends queries from August 2021 to August 2026 were each submitted once, giving 54 responses.
- Potentially harmful information appeared in 4 of 54 responses (7.4%), all involving antimicrobial recommendations.
- JAMA transparency scores were 0 in 52 of 54 responses (96.3%).
- None of the 54 responses met sixth-grade readability benchmarks.
- Copilot had the highest median EQIP score at 70.00, but no pairwise difference between models stayed significant after Holm correction.
The full story
Responses were scored with DISCERN, EQIP, JAMA and GQS quality tools plus six readability indices. Differences between models in DISCERN, EQIP and the readability indices were tested with Friedman tests, followed by paired Wilcoxon signed-rank tests with Holm correction.
All of those measures showed significant overall differences between models, but no individual model-to-model comparison stayed significant after correction. JAMA and GQS were only summarized descriptively, and harmful misinformation was assessed exploratorily.
The authors conclude the chatbots shared poor source transparency and excessive reading complexity. They say the tools may supplement general patient information but should not replace professional assessment or guideline-based care.
Sources
1 on file- Frontiers in Medicinedoi:10.3389/fmed.2026.1991323
Talk it over
Four of the 54 chatbot responses had potentially harmful information, all on antimicrobial recommendations, while none met sixth-grade reading benchmarks. How would you weigh the harmful advice against the reading difficulty?
Test of six AI chatbots finds 4 of 54 epididymitis answers had potentially harmful advicehttps://truvace.com/item/quality-readability-and-potential-safety-risks-of-epididymitis-information-gener
Four of the 54 chatbot responses had potentially harmful information, all on antimicrobial recommendations, while none met sixth-grade reading benchmarks. How would you weigh the harmful advice against the reading difficulty?
Answer in the threadEach of the nine queries went to ChatGPT, Gemini, Perplexity, Copilot, Kimi and DeepSeek once. How much can a single submission per query tell us about the advice a patient would get on another day?
Answer in the threadCopilot had the highest median EQIP score at 70.00, yet no pairwise difference between models stayed significant after Holm correction. What finding would make you confident that one chatbot is safer than another?
Answer in the thread
If you have more time
The nine queries came from 25 Google Trends queries. Which questions would a person with epididymitis symptoms ask that these nine might miss, and how could that change the results?
Answer in the threadJAMA benchmark scores were 0 in 52 of 54 responses. What would a chatbot need to show about its sources and authors before you trusted its medical advice, and who should check it?
Answer in the thread
ace
What readers say