Truace
Subscribe
60/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Screening tool saved a median 26 hours per review but missed some studies, 16-review study finds

Study title: Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews

Evidence type
Observational study
Sample
n = 802
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2818cite or check it

Study title: Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews

Researchers at an evidence synthesis centre tested Abstrackr, a free machine learning tool, on 16 of their own reviews (11 systematic, 5 rapid). After a reviewer screened 200 training records, they simulated letting the tool's predictions replace part of the second reviewer's work. The simulation saved a median 26 hours of screening per review and wrongly excluded 0 to 3 studies per review.

Read the study abstract

BACKGROUND: We evaluated the benefits and risks of using the Abstrackr machine learning (ML) tool to semi-automate title-abstract screening and explored whether Abstrackr's predictions varied by review or study-level characteristics. METHODS: For a convenience sample of 16 reviews for which adequate data were available to address our objectives (11 systematic reviews and 5 rapid reviews), we screened a 200-record training set in Abstrackr and downloaded the relevance (relevant or irrelevant) of the remaining rec…

Decoding semi-automated title-abstract screening: findings from a convenience sample of reviews
"Parkes Radio Telescope (CSIRO)" by Amanda Slater, CC BY-SA 2.0.

58Gain side

How the two sides scoreGain scores higher

53Problem side

The gain

In a retrospective simulation on 16 evidence reviews, using Abstrackr's predictions with liberal-accelerated screening saved a median 26 hours of title-abstract screening per review (35% of records) compared with dual independent screening.

The problem

In the same 16-review simulation of Abstrackr, the tool wrongly excluded 0 to 3 studies per review (0 to 14% of those included), and 13% of 802 included studies were predicted irrelevant, which changed one meta-analysis from significant to non-significant.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Time savings matter because title-abstract screening is one of the slowest steps in producing a systematic review. The tool correctly flagged 87% of the 802 studies that ended up in the final reports, and in six of ten systematic reviews it missed fewer studies than a single human reviewer would have.

The risks are uneven. Predictions were more accurate for trials (91%) than observational studies (79%) or reviews (83%), and only one of eight affected meta-analyses changed significance. The authors call several findings paradoxical and say prospective studies are needed.

Main points

  1. The sample was 16 reviews from one centre, 11 systematic reviews and 5 rapid reviews, chosen because adequate data were available.
  2. A 200-record training set was screened in Abstrackr, and its hard relevant/irrelevant predictions were used to simulate liberal-accelerated screening against dual independent screening.
  3. Median workload savings were 3,143 records (35%), equal to 26 hours or about 3 working days; no studies were missed in 5 of 10 systematic reviews.
  4. Of 802 included studies, 696 (87%) were correctly predicted relevant and 106 (13%) were wrongly predicted irrelevant.
  5. Accuracy was 95% in reviews of only trials versus 86% in reviews with multiple designs, and 89% with multiple research questions versus 83% with a single question.
  6. One missed antipsychotic trial fed eight meta-analyses; only the compulsions result changed, from a significant reduction (MD -1.53) to a non-significant one (MD -1.17).
  7. The simulation beat single-reviewer screening in 6 of 10 systematic reviews, for example 1% versus 11% missed in the activity and pregnancy review.

The full story

Records were screened in Abstrackr using random order and a single-reviewer setting, replicating the senior reviewer's original decisions for the 200-record training set. Predictions usually arrived within 24 hours. Time savings were estimated at 0.5 minutes per record and an 8-hour day.

In the simulation, the senior reviewer screened only records predicted relevant and the second reviewer checked only those excluded by either the senior reviewer or the tool. Half of the systematic reviews lost no studies, two lost one record (1%) and three lost three records (2 to 14%). A missed Chinese-language study with an English abstract was one of them.

Of 49 antipsychotic pairwise comparisons with at least low-strength evidence, one missed study appeared in 8 (16%), and only one changed statistical significance. The authors say removing missed studies would not have changed the main conclusions, and that comprehensive searches would likely catch such studies by other means.

Accuracy differed by study traits: trials 91%, qualitative studies 93%, observational studies 79%. Studies at unclear or high risk of bias were predicted correctly more often (88%) than low-risk-of-bias studies (80%), and the authors call this and other findings paradoxical. The approach saved slightly more time than a semi-automated second-screener method (20 hours versus 26) with identical missed records.

What this doesn’t fix

The 16 reviews were a convenience sample, the protocol was not published in advance, only one tool and one screening approach were tested, and only eight meta-analyses in one review were affected by missed studies, so the authors caution against generalising.

Sources

1 on file
  1. Systematic Reviewsdoi:10.1186/s13643-020-01528-x
    Peer-reviewed2020-11-26

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Screening tool saved a median 26 hours per review but missed some studies, 16-review study findshttps://truvace.com/item/decoding-semi-automated-title-abstract-screening-findings-from-a-convenience-sam

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a retrospective simulation on 16 evidence reviews, using Abstrackr's predictions with liberal-accelerated screening saved a median 26 hours of title-abstract screening per review (35% of records) compared with dual independent screening.

    The problem In the same 16-review simulation of Abstrackr, the tool wrongly excluded 0 to 3 studies per review (0 to 14% of those included), and 13% of 802 included studies were predicted irrelevant, which changed one meta-analysis from significant to non-significant.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on an observational study with a sample of 802, from Systematic Reviews.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The 16 reviews were a convenience sample, the protocol was not published in advance, only one tool and one screening approach were tested, and only eight meta-analyses in one review were affected by missed studies, so the authors caution against generalising.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Science