TRV-2026-2980Version 1 · Published

Saved · see the current record

Why this version exists

Certified into the record

The exact text we saved

This is the version exactly as it was saved, character for character. The SHA-256 hash below is a 64-character code worked out from that text. Run the same text through SHA-256 and you get the same code; change a single letter and the code comes out completely different. Each version also stores the previous version’s hash, which chains them together so no version can be swapped out quietly.

TRUVACE RECORD VERSION
record: TRV-2026-2980
version: 1
kind: certified
reason: Certified into the record
timestamp: 2026-11-17T00:02:19.840216Z
status: published
lens: trace
sector: health
headline: Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study
dek: Background: Health systems are increasingly deploying patient-facing conversational AI platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement.
Objective: This study aimed to evaluate a structured red teaming methodology for the safety assessment of a patient-facing clinical AI platform and determi…
gain_title: In a 2-phase simulation study at one academic health system, a patient-facing clinical AI platform's high-risk failure rate fell from 30.2% (35/116 transcripts) to 8.5% (20/234) after targeted remediation, a 21.7-point drop.
problem_title: In the first phase of a simulation study with 48 health care professionals at one academic health system, a patient-facing clinical AI platform failed 30.2% (35/116) of high-risk transcripts versus 5.6% (7/126) of medium-risk ones.
trace_subject: Safety failures of a patient-facing clinical AI platform in high-risk scenarios before and after remediation
gain_reading: In a 2-phase simulation study at one academic health system, a patient-facing clinical AI platform's high-risk failure rate fell from 30.2% (35/116 transcripts) to 8.5% (20/234) after targeted remediation, a 21.7-point drop.
gain_evidence: In phase 2, the high-risk failure rate was 8.5% (20/234), representing an absolute difference of 21.7 percentage points
problem_reading: In the first phase of a simulation study with 48 health care professionals at one academic health system, a patient-facing clinical AI platform failed 30.2% (35/116) of high-risk transcripts versus 5.6% (7/126) of medium-risk ones.
problem_evidence: the high-risk failure rate was 30.2% (35/116) of transcripts vs 5.6% (7/126) for medium-risk scenarios
quick_read: Researchers at a single academic health system tested a patient-facing conversational AI platform in January and February 2026. In phase 1, 30.2% of high-risk transcripts (35 of 116) were scored as failures, against 5.6% (7 of 126) for medium-risk scenarios.

After the platform developer fixed problems based on aggregated, deidentified phase 1 findings, 27 new participants ran 8 targeted high-risk scenarios. The high-risk failure rate fell to 8.5% (20 of 234), a 21.7-percentage-point absolute drop (participant-clustered 95% CI 11.1-32.2) and a 72% relative drop.

Remaining failures were not eliminated: they clustered around triage calibration errors and scope violations. Rater agreement on scoring was moderate in phase 2, so the improvement is an association measured in simulation rather than proof of safety in real patient use.
limitation: The work was a simulation at a single academic health system, using health care professionals rather than patients, and residual failures in triage calibration and scope remained after remediation.
tag: Dual reading
key_points: The study used 2 phases: 48 health care professionals completed 6 randomized scenarios (3 medium risk, 3 high risk) in phase 1, and 27 new participants completed 8 targeted high-risk scenarios in phase 2. | All 476 transcripts were independently scored by 3 calibrated raters using a 5-domain safety rubric. | Phase 1 high-risk failure rate was 30.2% (35/116) versus 5.6% (7/126) for medium-risk scenarios. | Phase 2 high-risk failure rate was 8.5% (20/234), with comparable performance on prespecified and improvised prompts within the targeted failure categories. | Residual failures clustered around triage calibration errors and scope violations.
rundown: The study was a 2-phase prospective simulation at one academic health system. A strict firewall separated the research team from the platform developer, and only aggregated, deidentified findings were passed between phases, so the developer's remediation was done independently.

Scoring relied on 3 calibrated raters applying a 5-domain safety rubric to all 476 transcripts. Intraclass correlation was 0.66 (95% CI 0.62-0.69) in phase 1 and 0.49 (95% CI 0.43-0.55) in phase 2, while agreement on the binary failure call held steady (Fleiss kappa 0.70 in both phases) and observed agreement rose from 90.9% (220/242) to 94.9% (222/234).

The authors conclude that structured iterative red teaming was associated with better safety on new high-risk scenarios after targeted remediation, and say the rubric and scenario taxonomy can be reused by health systems for predeployment and ongoing clinical AI safety assessment.
sources:
- peer_reviewed | JMIR AI | https://doi.org/10.2196/101912 | 2026-11-16
prev: 0000000000000000000000000000000000000000000000000000000000000000
SHA-256 hash
2f2521c90e39b2d609b0b80d5818585138a3b00617bdbbc64de88150ef882b48
Previous version’s hash
0000000000000000000000000000000000000000000000000000000000000000
Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2980 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.