Truace
Subscribe

LLM prompting strategies for systematic literature review screening

Source article: Evaluating prompting strategies and large language models in systematic literature review screening: Relevance and task-stage classification

This study quantifies how prompting strategies interact with large language models (LLMs) to automate the screening stage of systematic literature reviews (SLRs). We evaluate six LLMs (GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku, and Llama-4-Maverick) under five prompt types (zero-shot, few-shot, chain-of-thought [CoT], CoT-few-shot, and self-reflection) across relevance classification and six Level-2 tasks, using accuracy, precision, recall, and F1. Results show pronounced model-pr…

By the Truvace desk1 min read1 sourcePeer-reviewedTRV-2026-1935· cite & verify
Evaluating prompting strategies and large language models in systematic literature review screening: Relevance and task-stage classification
The development of an attitude measurement device for improvement of selection/ screening of U.S. personnel for overseas duty. by Mozingo, Thomas Phillip.. Public domain
Trace impact readingContested
P 64The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 67The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers compared six large language models under five prompt styles on relevance classification and six further screening tasks in systematic literature reviews. They scored each pairing on accuracy, precision, recall and F1, and also estimated cost per 1,000 abstracts.

The work matters because screening is a laborious stage of evidence synthesis, and the results offer practical guidance on matching model and prompt to the goal. The authors describe the potential as uneven but promising, and say further validation is needed before the findings are generalised beyond the corpus studied.

Main points

  1. Six LLMs were tested: GPT-4o, GPT-4o-mini, DeepSeek-Chat-V3, Gemini-2.5-Flash, Claude-3.5-Haiku and Llama-4-Maverick.
  2. Five prompt types were compared: zero-shot, few-shot, chain-of-thought, CoT-few-shot and self-reflection.
  3. Zero-shot maximized recall, while CoT-few-shot gave stronger precision and F1.
  4. GPT-4o and DeepSeek showed robust overall performance, and GPT-4o-mini was competitive at far lower cost.
  5. Self-reflection was over-inclusive and unstable across models.

The gain

Some model-prompt pairings screen abstracts effectively at low cost, with zero-shot maximizing recall and CoT-few-shot giving stronger precision and F1, while GPT-4o-mini stays cheap.

The problem

Self-reflection prompting performed poorly because it was over-inclusive and unstable across models, so results are uneven and only validated on one corpus.

The rundown

The evaluation covered relevance classification plus six Level-2 tasks, using accuracy, precision, recall and F1. It found pronounced interaction effects between model and prompt within the SLR-automation corpus.

A cost-performance analysis for relevance classification, per 1,000 abstracts, showed large absolute differences between pairings. GPT-4o-mini stayed low-cost across prompts, and structured prompts on it gave attractive F1 for a small added cost.

The authors suggest future workflows could be staged and cost-aware. Lower-cost, high-recall configurations would handle initial screening, and higher-cost or more precision-oriented configurations would be reserved for later triage or uncertain records.

What this doesn’t fix

Results come from a single SLR-automation corpus, and the authors state further validation is needed before generalising to broader systematic review screening tasks.

Sources

  1. Peer-reviewedResearch Synthesis Methods2026-10-19

The debate