Moderate evidence
More
Study finds red-team testing cut a patient AI platform's high-risk failures from 30.2% to 8.5%
Study title: Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study
- Evidence type
- Controlled or validated comparison
- Sample
- n = 476
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2980cite or check it
Study title: Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study
Researchers at a single academic health system tested a patient-facing conversational AI platform in January and February 2026. In phase 1, 30.2% of high-risk transcripts (35 of 116) were scored as failures, against 5.6% (7 of 126) for medium-risk scenarios.
Read the study abstract
Background: Health systems are increasingly deploying patient-facing conversational AI platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement. Objective: This study aimed to evaluate a structured red teaming methodology for the safety assessment of a patient-facing clinical AI platform and determi…
60Gain side
How the two sides scoreClose call
61Problem side
The gain
In a 2-phase simulation study at one academic health system, a patient-facing clinical AI platform's high-risk failure rate fell from 30.2% (35/116 transcripts) to 8.5% (20/234) after targeted remediation, a 21.7-point drop.
The problem
In the first phase of a simulation study with 48 health care professionals at one academic health system, a patient-facing clinical AI platform failed 30.2% (35/116) of high-risk transcripts versus 5.6% (7/126) of medium-risk ones.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
In brief
After the platform developer fixed problems based on aggregated, deidentified phase 1 findings, 27 new participants ran 8 targeted high-risk scenarios. The high-risk failure rate fell to 8.5% (20 of 234), a 21.7-percentage-point absolute drop (participant-clustered 95% CI 11.1-32.2) and a 72% relative drop.
Remaining failures were not eliminated: they clustered around triage calibration errors and scope violations. Rater agreement on scoring was moderate in phase 2, so the improvement is an association measured in simulation rather than proof of safety in real patient use.
Main points
- The study used 2 phases: 48 health care professionals completed 6 randomized scenarios (3 medium risk, 3 high risk) in phase 1, and 27 new participants completed 8 targeted high-risk scenarios in phase 2.
- All 476 transcripts were independently scored by 3 calibrated raters using a 5-domain safety rubric.
- Phase 1 high-risk failure rate was 30.2% (35/116) versus 5.6% (7/126) for medium-risk scenarios.
- Phase 2 high-risk failure rate was 8.5% (20/234), with comparable performance on prespecified and improvised prompts within the targeted failure categories.
- Residual failures clustered around triage calibration errors and scope violations.
The full story
The study was a 2-phase prospective simulation at one academic health system. A strict firewall separated the research team from the platform developer, and only aggregated, deidentified findings were passed between phases, so the developer's remediation was done independently.
Scoring relied on 3 calibrated raters applying a 5-domain safety rubric to all 476 transcripts. Intraclass correlation was 0.66 (95% CI 0.62-0.69) in phase 1 and 0.49 (95% CI 0.43-0.55) in phase 2, while agreement on the binary failure call held steady (Fleiss kappa 0.70 in both phases) and observed agreement rose from 90.9% (220/242) to 94.9% (222/234).
The authors conclude that structured iterative red teaming was associated with better safety on new high-risk scenarios after targeted remediation, and say the rubric and scenario taxonomy can be reused by health systems for predeployment and ongoing clinical AI safety assessment.
What this doesn’t fix
The work was a simulation at a single academic health system, using health care professionals rather than patients, and residual failures in triage calibration and scope remained after remediation.
Sources
1 on file- JMIR AIdoi:10.2196/101912
Talk it over
High-risk transcripts failed 30.2% of the time (35 of 116) in phase 1 and 8.5% (20 of 234) in phase 2. How would you weigh that improvement against the failures that remained?
Study finds red-team testing cut a patient AI platform's high-risk failures from 30.2% to 8.5%https://truvace.com/item/red-teaming-a-patient-facing-clinical-ai-platform-2-phase-prospective-study
High-risk transcripts failed 30.2% of the time (35 of 116) in phase 1 and 8.5% (20 of 234) in phase 2. How would you weigh that improvement against the failures that remained?
Answer in the threadThe 48 health care professionals played patients in a simulation at one academic health system. How might their prompts and behaviour differ from those of real patients, and how could that change the failure rates?
Answer in the threadIf you were the platform developer receiving only aggregated, deidentified phase 1 findings, what would you want to know before deciding what to fix first?
Answer in the thread
If you have more time
Rater agreement on scoring was moderate in phase 2 (intraclass correlation 0.49), while agreement on the binary failure call held at Fleiss kappa 0.70. What would you look up to judge how much that matters?
Answer in the threadThe authors say health systems can reuse the rubric and scenario taxonomy. How might a hospital or clinic you know decide which scenarios count as high-risk, given that triage calibration errors and scope violations remained?
Answer in the thread
ace
What readers say