68/100

Moderate evidence

Trace: a gain and a problemPublished recordHealthThe Trace

Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study

Truvace record
TRV-2026-2980raw data (JSON)
Version
v1Published
First published
Evidence type
Controlled or validated comparison
Sources
1 sourcePeer-reviewed
sha256
2f2521c90e39…check it

Background: Health systems are increasingly deploying patient-facing conversational AI platforms, yet validated methodologies for safety evaluation under realistic high-risk conditions are lacking. Prior work has documented safety failures in clinical AI but not shown whether iterative, health system-embedded testing can produce measurable safety improvement. Objective: This study aimed to evaluate a structured red teaming methodology for the safety assessment of a patient-facing clinical AI platform and determi…

60Gain side

How the two sides scoreClose call

61Problem side

The gain

In a 2-phase simulation study at one academic health system, a patient-facing clinical AI platform's high-risk failure rate fell from 30.2% (35/116 transcripts) to 8.5% (20/234) after targeted remediation, a 21.7-point drop.

The problem

In the first phase of a simulation study with 48 health care professionals at one academic health system, a patient-facing clinical AI platform failed 30.2% (35/116) of high-risk transcripts versus 5.6% (7/126) of medium-risk ones.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

What this doesn’t fix

The work was a simulation at a single academic health system, using health care professionals rather than patients, and residual failures in triage calibration and scope remained after remediation.

Sources

1 on file
  1. JMIR AIdoi:10.2196/101912
    Peer-reviewed2026-11-16

Cite this record

v1

Truvace Impact Record TRV-2026-2980, v1: “Red Teaming a Patient-Facing Clinical AI Platform: 2-Phase Prospective Study.” Truvace, 2026-11-17. /record/TRV-2026-2980 (accessed on the day you copy it). sha256 2f2521c90e39b2d6…

Change history

1 version

Every change made to this record since it was first published, and why. Nothing has changed since it went up.

  1. Publishedv12f2521c90e39…

    Certified into the record

Check this record hasn’t been altered

Check this record hasn’t been altered
How to check it yourself, without trusting this page

Download any version from /api/record/TRV-2026-2980 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.