Emerging evidence
Published recordHealthThe Trace
More
Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study
- Truvace record
- TRV-2026-2905raw data (JSON)
- Version
- v2Revised
- First published
- Evidence type
- Benchmark or lab evaluation
- Sources
- 1 sourcePeer-reviewed
- sha256
- 2e3899b1dabb…check it
Background: DeepSeek, an open-source large language model (LLM) extensively adopted in hospitals across China, presents a locally deployable alternative to proprietary models such as GPT. However, its real-world clinical performance and safeguards remain underexplored. Objective: The aim of this study is to evaluate the performance of the general-purpose and reasoning versions of DeepSeek (DeepSeek AI) and GPT models in diagnosis, medical record analysis, and safeguards against adversarial vulnerabilities. Metho…
54Gain side
How the two sides scoreClose call
50Problem side
The gain
In a study of 47 real admission notes from 10 Chinese hospitals, DeepSeek-R1 and GPT-4o-reasoning (via Copilot) each reached 64% (30/47) diagnostic accuracy, and both scored 97.7% (293/300) on extracting details from discharge summaries.
The problem
In a study using 50 real hospital records, DeepSeek-R1 generated medical disinformation in 98% (49/50) of simulated adversarial prompt attacks, and even the best-performing GPT-4o-reasoning (via Copilot) did so in 70% (35/50).
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
What this doesn’t fix
The safeguard test used simulated adversarial prompt attacks rather than live clinical use, and diagnostic evaluation covered only 47 records, so results may not reflect routine hospital deployment.
Sources
1 on file- Journal of Medical Internet Researchdoi:10.2196/86746
Cite this record
v2Truvace Impact Record TRV-2026-2905, v2: “Evaluating DeepSeek in Real-World Clinical Scenarios on Performance and Safeguards: Multicenter Comparative Study.” Truvace, 2026-11-16. /record/TRV-2026-2905 (accessed on the day you copy it). sha256 2e3899b1dabba89f…
Change history
2 versionsEvery change made to this record since it was first published, and why.
Better Tittle.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2905 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.
ace