64/100

Moderate evidence

Cite
More
EmailLinkedInX

AI reviewers showed no clear author bias in 50 abstracts, but scores may be too uniform

Study title: Bias and Reliability of AI-Based Peer Review: A Comparative Study of ChatGPT and Claude Evaluating Scientific Abstracts

Evidence type
Controlled or validated comparison
Sample
n = 50
Sources
1 sourcePeer-reviewed
Published
1 min read

Study title: Bias and Reliability of AI-Based Peer Review: A Comparative Study of ChatGPT and Claude Evaluating Scientific Abstracts

In April 2026, researchers tested ChatGPT and Claude as peer reviewers by having each model score 50 general internal medicine abstracts twice under four fictional author identities, producing 800 evaluations of quality, novelty, and acceptance on a 0-10 scale.

Read the study abstract

Background The use of artificial intelligence (AI) models as reviewers of scientific content raises concerns about potential biases related to author identity and about the reproducibility of their evaluations. We assessed whether AI-based reviewers exhibit gender or geographic bias and evaluated the reproducibility of their scoring of scientific abstracts. Methods We randomly selected 10 general internal medicine journals indexed in the Journal Citation Reports (impact factor ≥ 1.5). For each journal, five orig…

The gain

In a study of 800 evaluations of 50 medical abstracts under four fictional author identities, ChatGPT and Claude showed no consistent gender or geographic bias and scored with high reproducibility.

In brief

By the October 2026 publication date, the study reported no consistent gender or geographic bias and high reproducibility, but noted that narrow score distributions may have inflated agreement and that comparison to human peer review validity was still needed.

Main points

  1. 50 original research abstracts from 10 general internal medicine journals were each assigned to four fictional identities: African female, African male, American female, American male.
  2. Each abstract was evaluated twice by ChatGPT and Claude on quality, novelty, and acceptance on a 0-10 scale, yielding 800 evaluations in April 2026.
  3. ChatGPT quality and novelty scores were identical across identities with median 7 [1] and 5 [2], and Claude scores were identical across identities for all three dimensions.

The full story

The authors retrieved five original research abstracts from each of 10 general internal medicine journals indexed in Journal Citation Reports with impact factor ≥ 1.5, and created four fictional author identities per abstract. Bias was tested with multivariable ordinal logistic regression, and reproducibility with percent agreement and Fleiss' kappa.

Results showed median quality 7 [1], novelty 5 [2], and acceptance 6-7 [1-2] with minimal variation across identities, and multivariable analyses showed no overall association except for acceptance for Claude at the global level with no individual comparisons statistically significant.

Sources

1 on file
  1. Journal of General Internal Medicinedoi:10.1007/s11606-026-10806-8
    Peer-reviewed2026-10-02

Talk it over

Three questions to start on, two more if you have time
The study ran 800 evaluations of 50 abstracts under four fictional author identities and found no consistent gender or geographic bias. How far does that result go toward showing ChatGPT and Claude are fair reviewers?

Start with a question

AI reviewers showed no clear author bias in 50 abstracts, but scores may be too uniformhttps://truvace.com/item/bias-and-reliability-of-ai-based-peer-review-a-comparative-study-of-chatgpt-and-

  1. The study ran 800 evaluations of 50 abstracts under four fictional author identities and found no consistent gender or geographic bias. How far does that result go toward showing ChatGPT and Claude are fair reviewers?

    Answer in the thread
  2. The researchers noted that narrow score distributions may have inflated agreement. How would you weigh high reproducibility against the chance that the scores were too uniform to tell abstracts apart?

    Answer in the thread
  3. The abstracts came from 10 general internal medicine journals, and the identities were African female, African male, American female and American male. Which authors or kinds of research are missing, and how could that change the result?

    Answer in the thread
If you have more time
  • Look up how human peer review scores the same 50 abstracts. What comparison would settle whether the AI reviewers are valid and not only consistent?

    Answer in the thread
  • If a journal you know used ChatGPT or Claude to screen submissions, which decisions would you keep with human editors, and why?

    Answer in the thread

What readers say