Canonical text (the exact bytes fingerprinted)

TRUVACE RECORD VERSION
record: TRV-2026-2226
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-10-24T00:00:00.493148Z
status: published
lens: trace
sector: health
headline: Accuracy of Large Language Model Chatbots in Retrieving MRI Safety Conditions for Medical Implants
dek: Background Retrieving MRI safety conditions for implanted devices is increasingly complex and time-consuming. Purpose To evaluate the accuracy of internet-enabled large language model (LLM) chatbots in retrieving implant-specific MRI safety conditions, using manufacturer documentation as the reference standard. Study type Prospective comparative evaluation in a constructed device cohort, followed by a retrospective clinical pilot. Population Fifty purposively selected device configurations and 20 patients (10 wi…
gain_title: ChatGPT correctly retrieved about 94% of implant MRI safety parameters in a constructed cohort and 98% in a small clinical pilot, which could ease a time-consuming lookup task.
problem_title: Even the best-performing chatbot made errors, and Gemini and Claude were accurate on only about 78-79% of parameters, so outputs need checking against manufacturer documentation before clinical use.
trace_subject: LLM chatbot retrieval of MRI safety conditions for medical implants
gain_reading: ChatGPT correctly retrieved about 94% of implant MRI safety parameters in a constructed cohort and 98% in a small clinical pilot, which could ease a time-consuming lookup task.
gain_evidence: Internet-enabled LLM chatbots correctly retrieved up to 94% of MRI safety parameters in the constructed cohort and 98% in the clinical pilot.
problem_reading: Even the best-performing chatbot made errors, and Gemini and Claude were accurate on only about 78-79% of parameters, so outputs need checking against manufacturer documentation before clinical use.
problem_evidence: Errors occurred even with the best-performing chatbot, requiring direct verification against current manufacturer documentation before clinical use.
quick_read: Researchers tested three internet-enabled chatbots (ChatGPT-5.0, Gemini 2.5 Pro and Claude Sonnet 4.5) on eight MRI safety parameters for 50 implant device configurations. Manufacturer instructions and technical manuals served as the reference standard, and three experienced reviewers scored 1200 outputs. ChatGPT scored highest, with pooled accuracy of 0.94 versus 0.79 for Gemini and 0.78 for Claude. A follow-up pilot with a newer ChatGPT version on 20 referred patients reached 0.98.

The findings suggest chatbots could speed up a complex, time-consuming lookup that MRI safety teams face with implanted devices. But errors remained even for the top performer, so the results do not support using chatbot answers without checking the manufacturer sources. How the tools would perform across wider device types and in routine clinical workflows is not established by this study.
limitation: Devices were purposively selected, the clinical pilot was retrospective and involved only 20 patients and a newer ChatGPT version, and errors persisted even with the best chatbot.
tag: Dual reading
key_points: The prompts asked for eight parameters, including the MRI safety label, field strengths, RF exposure limits, maximum spatial gradient and scan duration. | Mean ordinal scores were 4.76 for ChatGPT, 4.20 for Gemini and 4.17 for Claude, and ChatGPT's lead was statistically significant. | Pooled accuracy was 0.94 for ChatGPT, 0.79 for Gemini and 0.78 for Claude, using scores of 4-5 as accurate. | The clinical pilot scored 160 outputs from ChatGPT-5.6 for 20 patients, 10 with cardiac and 10 with noncardiac devices, and found 0.98 accuracy.
rundown: The study had two stages: a prospective comparison on 50 purposively selected device configurations, followed by a retrospective clinical pilot. The pilot drew on patients referred to the MRI Safety Committee. Each chatbot received standardized single-shot prompts with no field-strength or sequence restrictions.

Two radiologists with 10 and 25 years of experience and an MRI technologist with 32 years independently scored the outputs on a 5-point scale. A score of 5 meant fully correct and 1 meant substantively incorrect or fabricated. Agreement was assessed with the intraclass correlation coefficient and Gwet's AC1, and confidence intervals came from a device-clustered bootstrap.

The authors rated the evidence as Stage 2 technical efficacy. They concluded that direct verification against current manufacturer documentation is required before clinical use.
sources:
- peer_reviewed | Journal of Magnetic Resonance Imaging | https://doi.org/10.1002/jmri.70578 | 2026-10-21
prev: 0000000000000000000000000000000000000000000000000000000000000000

sha256

d35f5a3311e1c3ed0ca160c1660543e59c03e1e55c2a8baaf0e00ef099e456cd

previous

0000000000000000000000000000000000000000000000000000000000000000