Truace
Subscribe

Four AI chatbots gave exact, real medical references for only 128 of 360 requests

Study title: Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study

Researchers asked ChatGPT, Grok, Meta AI and Perplexity to supply bibliographic references on three medical and health topics: aflatoxin toxicity, hospital sanitization and human papillomavirus vaccination. Each prompt was repeated three times per platform, and every reference was checked by hand against Scopus and Google Scholar.

Read the study abstract

Background: The use of generative AI in research and clinical practice is rapidly expanding, offering unprecedented capabilities in pattern extraction and data analysis. However, its application raises critical concerns regarding the accuracy and reliability of the information provided. Objective: The aim of this study is to analyze and compare the performance of 4 generative AI platforms (ChatGPT, Grok, Meta AI, and Perplexity) in retrieving bibliographic references within the medical and health care field and…

By the Truvace desk2 min read1 sourceEvidence: Moderate · 52/100 — Benchmark or lab evaluation · n = 360 · 1 independent sourcePeer-reviewedTRV-2026-2363· cite & verify
Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study
"Downtown Nassau - 2025 - Doctors Hospital (2)" by Bluerasberry is licensed under CC BY 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/.

In brief

The findings matter because researchers and clinicians increasingly use chatbots for literature searches, and invented citations can slip into medical work unnoticed. The results apply only to the specific prompts and guest-mode conditions tested, so they show a pattern of unreliability rather than a ranking that will hold for all uses or future model versions.

Main points

  1. Of 360 requested references, 273 were generated, a task completion rate of 75.8%.
  2. Of the 273 generated, 190 followed the requested article format, and 128 of those 190 (67.4%) matched real evidence exactly.
  3. Grok had the best result with 71 of 90 exact matches (78.9%), and ChatGPT had 46 of 90 (51.1%).
  4. Perplexity showed critical rates of total hallucination and could not separate scientific articles from gray literature.
  5. Meta AI failed the task systematically, with 0 of 10 exact matches in the case reported.

The problem

Across four AI chatbots, only 128 of 360 requested medical references matched real evidence exactly; 273 were generated, and Perplexity showed critical rates of total hallucination.

The rundown

The study was a cross-sectional observational comparison using three bibliographic queries of differing specificity. Each platform was used in Guest mode, and each prompt was run in triplicate. Accuracy was verified by hierarchical lookup that prioritized DOI.

Hallucinations were scored from 0 to 5 based on metadata correctness. The ability to provide real references depended significantly on the model used, and Kruskal-Wallis tests showed significant differences between models on every analyzed dimension.

Grok favored recent sources, with a median year of 2022, and had a mean hallucination score of 0.2. ChatGPT was stable but anchored to older, highly cited literature, with a median year of 2020 and a median of 73.5 citations.

The authors call for structured AI literacy programs in medical education. They recommend using such tools only as a complement, always subject to rigorous human fact-checking.

Sources

  1. Peer-reviewedJMIR Formative Research2026-11-03

The debate