Comparison of DeepSeek AI and Human Reviewers in Structured Evaluation of Neurosurgical Manuscripts: A Pilot Study

Objectives Increasing manuscript volume has strained peer review, prompting interest in artificial intelligence (AI) as a supportive tool. We evaluated DeepSeek R1 as a structured peer reviewer of neurosurgical case-based manuscripts and compared its scoring patterns with those of an experienced editor and student reviewers. Design Exploratory, cross-sectional comparative study using a convenience sample of consecutively submitted manuscripts. Setting Single skull-base neurosurgical journal during a 6-month subm…

By the Truvace deskOctober 15, 20262 min read1 sourcePeer-reviewedTRV-2026-1447· cite & verify

Comparison of DeepSeek AI and Human Reviewers in Structured Evaluation of Neurosurgical Manuscripts: A Pilot Study — "Cerebral Folate Deficiency - Cerebral CT-scan at 4 years old" by Sarah Mafi, Pierre-Antoine Faye is licensed under CC BY 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by/4.0/.

In brief

A skull-base neurosurgical journal tested DeepSeek R1 as a structured peer reviewer. Over a six-month window in 2025, the model, the journal's Editor-in-Chief and 154 first-year medical students each rated 11 de-identified, unpublished case-based manuscripts on a 3-point scale across five domains.

The study matters because rising manuscript volume strains peer review, and AI is being considered as support. The pattern of inflated subjective scores suggests the tool cannot replace expert judgment. The small sample and exploratory design leave open how the findings generalize to other journals or models.

Main points

Eleven manuscripts were assessed by three reviewer groups: 154 first-year medical students, the Editor-in-Chief, and DeepSeek R1.
Reviewers scored Study Objective, Novelty, Study Design, Thoroughness and Clarity on a 3-point scale.
Significant between-group differences appeared in four of five domains; Study Design did not differ (p = 0.483).
DeepSeek rated Novelty higher than both human groups and Study Objective higher than the experienced editor.
The authors propose a human-in-the-loop model combining AI efficiency with expert judgment.

The problem

In a pilot of 11 neurosurgical manuscripts, DeepSeek R1 gave systematically higher subjective ratings than human reviewers, notably for Novelty, which limits its reliability as an unsupervised peer reviewer.

The rundown

The design was an exploratory, cross-sectional comparison using a convenience sample of consecutively submitted manuscripts at a single skull-base neurosurgical journal between January and June 2025. Each student was randomly assigned one manuscript after a course in biostatistics, epidemiology and critical appraisal.

Kruskal-Wallis style statistics showed group differences for Study Objective (H = 8.31, p = 0.02), Novelty (H = 9.74, p = 0.008), Thoroughness (H = 10.49, p = 0.005) and Clarity (H = 8.13, p = 0.02). Study Design scoring was statistically similar across groups.

The authors describe DeepSeek as consistent in structured domains but prone to inflating subjective ones, and suggest a human-in-the-loop approach to peer review.