The test you wrote yourself

Every AI system that touches medicine performs a transformation. A conversation becomes a note. A scan becomes a label. A page of prose becomes a row in a spreadsheet. Something goes in, something smaller comes out, and the only question that really matters is whether the part that had to survive actually did.

This month I want to read October's records through a piece of my own work, because it failed in a way I think every reader of AI claims should understand.

A note is supposed to leave things out

In August, Kirsten Ostherr, Eivind Engebretsen and Angela Woods argued in The Lancet that ambient scribes are narrative technologies. They select, filter and reformulate what happens in the room. Their warning was that current approaches take "a superficial approach to narrative interpretation that omits the clinically relevant aspects of patient experience."

I agree with the warning. The difficulty is turning it into a measurement.

The obvious move, flagging everything the note left out, fails at once. Omission is the job of a note. A scribe that wrote down every word would have failed. So a fidelity measure cannot be built on absence. It needs a positive account of what must not be dropped.

That is what my Narrative Fidelity Framework tries to supply. It has three layers. The first is provenance: can each sentence in the note be traced to the exact words it rests on? The second is salience: four kinds of patient speech that only the patient can provide and that change what should happen next. A stated priority. A stated worry. The patient's own belief about what is causing the problem. A practical barrier, such as cost or transport. The third layer names six ways that speech can fail to survive. It can be dropped entirely. It can be credited to the wrong person. It can lose its uncertainty. A view can be recorded as a fact. Its meaning can change. Or the plan can act on it while the note never says why.

The paper's worked example, which I constructed for illustration, shows that last failure. A patient says the pharmacy wanted forty dollars last time and that paying it every month is not possible right now. The clinician switches the prescription to a generic. The note records the switch and not the reason. The care was good. The record of why it was good did not survive.

Ninety-four percent, then eight

Here is the part that matters for this month's reading.

I first built the framework against five clinical encounters I wrote myself, with planted salient content and a hand-built detector to find it. On that material the detector reached 94% precision and 79% recall. I never published those figures. I had written the encounters, the detector's patterns and the answer key. Nothing independent existed to say what the numbers were worth.

So I froze everything. The protocol, the prompts, the detector and the analysis plan were fixed by cryptographic hash before I read the held-out data. I wrote down decision rules in advance, including rules that committed me to reporting results against my own framework's interest. Then I ran it on 536 encounters and 21,902 patient turns from three public corpora that I did not author.

The detector fell to 8% precision and 4% recall. It missed 27 of the 28 turns the adjudication panel judged salient. One of the turns it never saw was a patient asking, "Will I be in a wheelchair?"

The framework narrowed further. Of the four kinds of salient speech, only one, stated worry, was recognised consistently by independent annotators on all three corpora. Agreement, measured with a statistic called Krippendorff's alpha, was .728, .677 and .866, against a bar of .67 that I set before looking. The other three categories did not clear it. Salient content was also rarer than my product premise assumed: in 23 of the 32 held-out consultations, the panel found nothing salient at all.

What survived was still useful. When I scored different note writers on the same encounters, they differed consistently in whether patients' stated worries made it into the record. In one consultation, a patient raised a fear of a brain tumour across five turns. The clinician's note recorded it in four words. One AI-drafted note, from gpt-4o, longer and better organised than the clinician's, did not mention the fear at all. Another configuration, using Grok-4.20, recorded the fear and the reason behind it.

The limits are real and I want them stated plainly. The salience judgments came from a panel of language models, not clinicians. The key comparison rests on 16 stated-worry turns. The scribes wrote from transcripts alone, which is not how any deployed product works. The consultations were role-played. The study does not establish clinical performance, documentation quality or patient benefit, and it says nothing about any deployed product, including mine.

The most transferable result is not about scribes at all. It is the distance between 94 and 8.

October's records, read through that gap

Truvace certified four health records in the first five days of October that show the same pressures from other directions. None of them is a scandal. Several are careful studies by authors who state their own limits. That is exactly why they are worth reading closely.

The first is a Trace, TRV-2026-1282. Researchers in the French ORFeAD forensic network tested two ways of extracting psychological symptoms from 110 forensic medical certificates written for victims of interpersonal violence. One pipeline used rules and word lists. The other used a locally run Llama 3 8B model. Both were scored against coding by forensic physicians. The authors note that these certificates describe symptoms in prose that is "rarely explicit and requiring inference."

The gain reading is real: specificity was high for most variables, and 24 of 35 variables met an 85% reliability threshold under at least one pipeline. The problem reading is equally real: sensitivity was low for low-prevalence symptoms, 11 variables failed under both pipelines, and the authors say neither pipeline supports decisions about an individual person. The language model did not clearly outperform the rules, at roughly three orders of magnitude more energy.

That is my framework's lesson in someone else's data. The content hardest to capture is rare, implicit and personal. The authors also refused to let one pooled score stand for the whole system. They require each variable to qualify on its own. That is the right instinct.

The second, TRV-2026-1260 in G Space, is a genuine gain. A team published in JAMIA Open fine-tuned LLaMA models on 1,427 annotated clinical notes from 95 patients to pull symptom presence and severity out of free text. Severity F1 scores ranged from 0.76 to 0.96, and symptom classification reached 0.99 for high-prevalence symptoms. The result was the first structured quarterly trajectories for 23 patients with vagus nerve stimulators.

Read the qualifier, though. The record notes uncertainty about reliability for low-prevalence psychiatric symptoms, and the paper itself reports that sensitivity for suicidal ideation was lower on the surgical comparison notes than on the stimulation notes, 0.80 against 1.00 for the largest model. The authors say this limits generalisation for psychiatric risk detection. The headline is carried by common symptoms. The rare one is the one you would most need it to catch.

The third, TRV-2026-1265 in P Space, is about what happens when a model leaves home. Researchers at Patan Hospital in Nepal ran a publicly available chest X-ray model in shadow mode on 826 consecutive health-assessment applicants. It still ranked images well, with an area under the curve of 0.958 and sensitivity of 95.1% at the chosen threshold. But its scores were compressed into a narrow band between 0.52 and 0.72. Positive predictive value was 17.9%, agreement with radiologists was low, and calibration failed.

The authors describe this as a marker of distributional shift in a low- and middle-income setting. They also say plainly that their threshold was chosen after the fact to reach the sensitivity they wanted, that independent external validation was not performed, and that local calibration is needed before deployment. A model can still sort cases in the right order while its numbers mean something different in a new place. And a threshold chosen after seeing the data is a fit, not a test.

The fourth is a Trace, TRV-2026-1283. Researchers built an XGBoost model to identify mild cognitive impairment among older adults with chronic conditions in the China Health and Retirement Longitudinal Study. On the 2018 development and internal validation data, discrimination was moderate and relatively stable, and the TreeSHAP explanations of which features drove the predictions held up across survey waves.

Then they moved the same model forward in time to the 2020 wave. Sensitivity fell to 0.390. The Brier score, a measure of how well predicted probabilities match outcomes, worsened to 0.137. The authors say the model needs recalibration and external validation before clinical use, and that it should not replace formal diagnosis or claim to predict future decline.

That is the same structure as my 94 and my 8. A number measured close to home, and a weaker one measured later, on a wave the builders had not used to tune the model. Truvace keeps the temporal validation reading for the same reason my study wrote its rules down first.

What October asks of a reader

None of these four records says AI fails in medicine. Two of them describe tools that may genuinely help. What they share is a structure: a strong number measured close to home, and a weaker or more qualified one measured further away.

When you meet an AI claim this month, I would ask four questions. Who wrote the test? Were the rules fixed before the results came in? What happened on data the builder never saw? And is the headline carried by the common cases, while the rare ones, the worry, the subtle finding, the symptom nobody names directly, quietly fall through?

My own detector scored 94% on my test and 8% on everyone else's. I would rather know that than not. The paper is public, and every label and frozen hash is deposited so that people better placed than I am can check it and disagree.

Whether a patient's fear survives into the record should not depend on which tool happened to be in the room. In my evaluation, under the limits above, it did. The answer is not hype or alarm. It is measurement on someone else's data, with the rules written down first.

Sources cited in this reading

Every factual claim in this reading is tied to a peer-reviewed publication or to my own openly deposited preprint, which is identified as such.

- Mosisa Saba, "The Narrative Fidelity Framework: A Preregistered Stress Test of Patient-Narrative Measures for Ambient Clinical Notes", Zenodo preprint, version v1, September 16, 2026, CC BY 4.0.

- Kirsten Ostherr, Eivind Engebretsen and Angela Woods, "Ambient scribes as narrative technologies", The Lancet, published online August 18, 2026.

- Laure Grisez et al., "Rule-Based Versus Generative Extraction of Psychological Symptoms From Forensic Medical Certificates: A Feasibility Study in the French ORFeAD Network", Behavioral Sciences & the Law, October 3, 2026.

- Vedansh Thakkar et al., "Longitudinal symptom severity tracking in vagus nerve stimulation patients: a 2-stage LLM-based pipeline with explainable AI", JAMIA Open, October 1, 2026.

- Lochan Shrestha et al., "Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal", BMJ Open, October 1, 2026.

- Kun Ruan and Yuan Li, "Explainable machine learning for identifying mild cognitive impairment in older adults with chronic diseases: Model development and temporal validation", Acta Psychologica, October 2026 (source date October 3, 2026).