Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2363
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-04T06:02:06.141298Z
status: published
lens: p_space
sector: health
headline: Comparative Evaluation of AI Chatbots for Medical Research: Cross-Sectional Observational Study
dek: Background: The use of generative AI in research and clinical practice is rapidly expanding, offering unprecedented capabilities in pattern extraction and data analysis. However, its application raises critical concerns regarding the accuracy and reliability of the information provided.
Objective: The aim of this study is to analyze and compare the performance of 4 generative AI platforms (ChatGPT, Grok, Meta AI, and Perplexity) in retrieving bibliographic references within the medical and health care field and…
gain_title: (none)
problem_title: Only 128 of 360 requested medical references (35.6%) matched real evidence, so the chatbots could not reliably replace validated bibliographic databases because of unstable answers and fabricated metadata.
trace_subject: (none)
gain_reading: (none)
gain_evidence: (none)
problem_reading: Only 128 of 360 requested medical references (35.6%) matched real evidence, so the chatbots could not reliably replace validated bibliographic databases because of unstable answers and fabricated metadata.
problem_evidence: resulting in an overall strict yield of 35.6% (128/360) from the initial requested references
quick_read: Researchers asked ChatGPT, Grok, Meta AI and Perplexity to supply bibliographic references on three medical and health topics: aflatoxin toxicity, hospital sanitization and human papillomavirus vaccination. Each prompt was repeated three times per platform, and every reference was checked by hand against Scopus and Google Scholar.

The findings matter because researchers and clinicians increasingly use chatbots for literature searches, and invented citations can slip into medical work unnoticed. The results apply only to the specific prompts and guest-mode conditions tested, so they show a pattern of unreliability rather than a ranking that will hold for all uses or future model versions.
limitation: The authors state that the findings apply only under the specific testing conditions of the study, which covered three queries run in Guest mode on four platforms.
tag: Evidence-backed problem
key_points: Of 360 requested references, 273 were generated, a task completion rate of 75.8%. | Of the 273 generated, 190 followed the requested article format, and 128 of those 190 (67.4%) matched real evidence exactly. | Grok had the best result with 71 of 90 exact matches (78.9%), and ChatGPT had 46 of 90 (51.1%). | Perplexity showed critical rates of total hallucination and could not separate scientific articles from gray literature. | Meta AI failed the task systematically, with 0 of 10 exact matches in the case reported.
rundown: The study was a cross-sectional observational comparison using three bibliographic queries of differing specificity. Each platform was used in Guest mode, and each prompt was run in triplicate. Accuracy was verified by hierarchical lookup that prioritized DOI.

Hallucinations were scored from 0 to 5 based on metadata correctness. The ability to provide real references depended significantly on the model used, and Kruskal-Wallis tests showed significant differences between models on every analyzed dimension.

Grok favored recent sources, with a median year of 2022, and had a mean hallucination score of 0.2. ChatGPT was stable but anchored to older, highly cited literature, with a median year of 2020 and a median of 73.5 citations.

The authors call for structured AI literacy programs in medical education. They recommend using such tools only as a complement, always subject to rigorous human fact-checking.
sources:
- peer_reviewed | JMIR Formative Research | https://doi.org/10.2196/103592 | 2026-11-03
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

002cd4edea1a9d085434330c91d581b630c17933348fe1666ebefdc5ea312a94

previous

0000000000000000000000000000000000000000000000000000000000000000