Review of 14 AI screening tools shows sensitivity near 90% for research papers

The authors also built their own screening tool on GPT-4o-mini and tested it on existing systematic reviews. It caught every relevant record (100% sensitivity) but had lower specificity (81%) and precision of 14%, and processed each entry in 1.7 seconds at minimal cost.

Screening is a labour-intensive, error-prone step in producing systematic reviews, so faster automated screening could cut the workload for research teams. The authors say the trade-off in specificity needs careful consideration before such tools are used.

Studies were included only if they reported enough data to build a confusion matrix; those requiring human intervention were excluded. Performance metrics were pooled with forest plots and summary receiver operating characteristic curves, and risk of bias was assessed with QUADAS-2.

The authors' own tool, built on GPT-4o-mini and tested against existing systematic reviews, missed no relevant records. Its precision of 14% and specificity of 81% mean many irrelevant records would still be passed on for human review.

The authors conclude that LLM-based models are a promising, cost-effective alternative to manual screening. They say integrating them could improve efficiency while maintaining reliability in academic research.