Truace
Subscribe

Open-source AI screened research records at 19.4 per minute versus about 1.0 for people

Study title: Lightweight open-source reasoning large language model for evidence synthesis screening

Researchers compared gpt-oss-20b, a lightweight open-source reasoning model, with two expert reviewers who screened subsets of records in three stages: base eligibility, obesity focus, and a focus on women of reproductive age. The model then screened all 20,118 records in the database.

Read the study abstract

Objectives: To evaluate performance and efficiency of a lightweight open-source reasoning large language model (LLM) against an expert-derived reference standard for screening scientific publication databases. Materials and methods: Two expert reviewers screened subsets of database records across 3 stages: (1) base eligibility (n = 200), (2) obesity focus (n = 200), and (3) women of reproductive age focus (n = 150). The LLM (gpt-oss-20b) screened all records (n = 20 118). Performance (F1-score and sensitivity [r…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-2240· cite & verify
Peer-reviewed study: JAMIA Open

In brief

The results suggest that smaller open models could ease one of the most labor-intensive steps in evidence synthesis. The authors describe the findings as promising rather than definitive, and the abstract reports performance only against the expert-derived reference subsets, not any real-world deployment.

Main points

  1. Two expert reviewers screened subsets of 200, 200, and 150 records across three screening stages.
  2. Interreader agreement between the experts was near-perfect, with kappa of 0.99-1.00.
  3. The model's F1-score ranged from 0.83 to 0.94, and its accuracy from 89% to 94%.
  4. Precision varied widely, from 77% to 99%, while specificity ranged from 86% to 99%.
  5. The model screened about 19.4 records per minute versus roughly 1.0 for human reviewers.

The gain

A 20-billion-parameter open-source reasoning model screened database records against an expert reference standard with 90%-95% sensitivity at about 19.4 records per minute, versus roughly 1.0 for human reviewers.

The rundown

The study tested the model at three screening stages with progressively narrower criteria: general eligibility, a focus on obesity, and a focus on women of reproductive age. Expert reviewers produced the reference standard on subsets of records at each stage.

Reported metrics included F1-score, sensitivity, specificity, accuracy, precision, and negative predictive value of 89%-96%. The spread in precision shows that performance differed across screening stages.

On efficiency, the model's average screening rate was about 19 times the human rate in the reported comparison. The authors conclude that lightweight open-source reasoning models show promise for supporting screening workflows.

Sources

  1. Peer-reviewedJAMIA Open2026-10-23

The debate