Moderate evidence
Published recordHealthThe Trace
More
Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study
- Truvace record
- TRV-2026-2980raw data (JSON)
- Version
- v1Published
- First published
- Evidence type
- Controlled or validated comparison
- Sources
- 1 sourcePeer-reviewed
- sha256
- 2f2521c90e39…check it
Background: Health systems are increasingly deploying patient-facing conversational AI platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement. Objective: This study aimed to evaluate a structured red teaming methodology for the safety assessment of a patient-facing clinical AI platform and determi…
60Gain side
How the two sides scoreClose call
61Problem side
The gain
In a 2-phase simulation study at one academic health system, a patient-facing clinical AI platform's high-risk failure rate fell from 30.2% (35/116 transcripts) to 8.5% (20/234) after targeted remediation, a 21.7-point drop.
The problem
In the first phase of a simulation study with 48 health care professionals at one academic health system, a patient-facing clinical AI platform failed 30.2% (35/116) of high-risk transcripts versus 5.6% (7/126) of medium-risk ones.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
What this doesn’t fix
The work was a simulation at a single academic health system, using health care professionals rather than patients, and residual failures in triage calibration and scope remained after remediation.
Sources
1 on file- JMIR AIdoi:10.2196/101912
Cite this record
v1Truvace Impact Record TRV-2026-2980, v1: “Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study.” Truvace, 2026-11-17. /record/TRV-2026-2980 (accessed on the day you copy it). sha256 2f2521c90e39b2d6…
Change history
1 versionEvery change made to this record since it was first published, and why. Nothing has changed since it went up.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2980 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.
ace