Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Review of 7 studies finds AI mammogram reading trades missed cancers for false alarms

Study title: Frequency and characteristics of errors by artificial intelligence (AI) in reading screening mammography: a systematic review

Evidence type
Systematic review or meta-analysis
Sample
n = 447,676
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2725cite or check it

Study title: Frequency and characteristics of errors by artificial intelligence (AI) in reading screening mammography: a systematic review

A systematic review of seven retrospective studies, covering 447,676 screening mammograms published between 2019 and 2022, quantified how often AI makes errors when reading breast screening images in real-world programmes. Across positivity thresholds of the Transpara tool, pooled false positives fell from 71.83% at threshold 3 to 10.77% at threshold 9, while pooled false negatives rose from 0.02% to 0.12%.

Read the study abstract

PURPOSE: Artificial intelligence (AI) for reading breast screening mammograms could potentially replace (some) human-reading and improve screening effectiveness. This systematic review aims to identify and quantify the types of AI errors to better understand the consequences of implementing this technology. METHODS: Electronic databases were searched for external validation studies of the accuracy of AI algorithms in real-world screening mammograms. Descriptive synthesis was performed on error types and frequenc…

Frequency and characteristics of errors by artificial intelligence (AI) in reading screening mammography: a systematic review
"An MRI scanner" by Walter Davies, CC BY-SA 4.0.

The problem

In a systematic review of seven studies covering 447,676 screening mammograms, the pooled false positive rate of the Transpara AI tool was 71.83% at threshold 3 and still 10.77% at threshold 9, and raising the threshold pushed missed cancers up to 0.12%.

In brief

The trade-off matters for screening programmes considering AI to replace some human reading. A lower threshold means many women flagged unnecessarily; a higher one means more missed cancers. Results varied within each threshold by algorithm version and by whether studies counted interval cancers, so the figures are not fixed properties of the tool.

Other kinds of error were almost never reported. Only one study addressed technical failures, and it found none; one study examined location errors, where AI marked the wrong place. The authors say this thin reporting limits understanding of how AI would affect screening workflow and outcomes.

Main points

  1. Seven retrospective studies with 447,676 examinations, published 2019 to 2022, met the inclusion criteria after 1,760 unique records were screened.
  2. Pooled false positives were 46.86% at Transpara threshold 5 and 29.86% at threshold 7, the most commonly reported triage thresholds.
  3. Studies counting both screen-detected and interval cancers showed lower false positive and higher false negative proportions than studies counting screen-detected cancers only.
  4. Among cancers the AI missed, 88% to 100% were invasive in two studies and 53% to 100% were interval cancers in two studies.
  5. Median tumour size of missed cancers ranged from 7 to 25 mm, versus 9 to 30 mm for cancers the AI detected in one registry-linked study.
  6. Four of the seven studies had high risk of bias or applicability concerns in at least one domain.
  7. In the one study of location errors, 8 of 18 wrongly placed AI markings were later confirmed as benign lesions.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • False negative proportion of AI reading screening mammograms at Transpara threshold 9

    Proportion 0.12% (95% CI 0.06% to 0.26%)

    The review’s own words
    The pooled FNP increased incrementally from 0.02% [95% CI 0.01, 0.03] at Transpara 3 to 0.12% [95% CI 0.06, 0.26] (Transpara 9), consistent with a trade-off with FPP.

  • False positive proportion of AI reading screening mammograms at Transpara threshold 3

    Proportion 71.83% (95% CI 69.67% to 73.90%)

    The review’s own words
    Pooled FPP decreased incrementally with increasing Transpara threshold (71.83% [95% CI 69.67, 73.90] at Transpara 3 to 10.77% [95% CI 8.34, 13.79] at Transpara 9).

  • False positive proportion of AI reading screening mammograms at Transpara threshold 9

    Proportion 10.77% (95% CI 8.34% to 13.79%)

    The review’s own words
    Pooled FPP decreased incrementally with increasing Transpara threshold (71.83% [95% CI 69.67, 73.90] at Transpara 3 to 10.77% [95% CI 8.34, 13.79] at Transpara 9).

  • False negative proportion of AI reading screening mammograms at Transpara threshold 3

    Proportion 0.02% (95% CI 0.01% to 0.03%)

    The review’s own words
    The pooled FNP increased incrementally from 0.02% [95% CI 0.01, 0.03] at Transpara 3 to 0.12% [95% CI 0.06, 0.26] (Transpara 9), consistent with a trade-off with FPP.

  • False positive proportion of AI reading screening mammograms at Transpara threshold 5

    Proportion 46.86% (95% CI 39.33% to 54.53%)

    The review’s own words
    The most commonly reported prospective triage thresholds were Transpara 5 (pooled FPP 46.86% [95% CI 39.33, 54.53])

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The review followed PRISMA-DTA guidelines and was registered in PROSPERO. It searched MEDLINE, EMBASE, SCOPUS and ArXiv for external validation studies in population screening where AI acted as an independent reader, either standalone or as a triage step. Five studies used versions 1.4.0 to 1.7.0 of Transpara, one used an ensemble model and one used a prototype AI-CAD system.

The datasets came from Sweden (two studies), Denmark, Norway, Germany, Spain and the United States, with enrolment from 2008 to 2018 and mean ages of 53 to 60. Cancer prevalence was 0.64% to 0.71% in studies with screen-detected cancers only and 0.71% to 1.22% where interval cancers were included.

Later Transpara versions (1.6.0 and 1.7.0) showed lower false negatives than version 1.4.0, but with more false positives, which the authors say could mean more recalls. Single radiologist sensitivity was comparable to Transpara score 7 (0.83 to 0.92) or 9 (0.77 to 0.88).

The authors recommend that future studies include all interval cancers, ideally through registry linkage, and report errors separately for screen-detected and interval cancers.

Sources

1 on file
  1. Breast Cancer Research and Treatmentdoi:10.1007/s10549-024-07353-3
    Peer-reviewed2024-06-09

Talk it over

Four questions to start onWho carries this cost, and what would have to change to prevent it?

Start with a question

Review of 7 studies finds AI mammogram reading trades missed cancers for false alarmshttps://truvace.com/item/frequency-and-characteristics-of-errors-by-artificial-intelligence-ai-in-reading

  1. Who carries this cost, and what would have to change to prevent it?

    The problem In a systematic review of seven studies covering 447,676 screening mammograms, the pooled false positive rate of the Transpara AI tool was 71.83% at threshold 3 and still 10.77% at threshold 9, and raising the threshold pushed missed cancers up to 0.12%.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 447,676, from Breast Cancer Research and Treatment.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health