Moderate evidence
Published recordScienceGood Space
More
Bias and Reliability of AI-Based Peer Review: A Comparative Study of ChatGPT and Claude Evaluating Scientific Abstracts
- Truvace record
- TRV-2026-1274raw data (JSON)
- Version
- v3Revised
- First published
- Evidence type
- Controlled or validated comparison
- Sources
- 1 sourcePeer-reviewed
- sha256
- 89eef90b48bf…check it
Background The use of artificial intelligence (AI) models as reviewers of scientific content raises concerns about potential biases related to author identity and about the reproducibility of their evaluations. We assessed whether AI-based reviewers exhibit gender or geographic bias and evaluated the reproducibility of their scoring of scientific abstracts. Methods We randomly selected 10 general internal medicine journals indexed in the Journal Citation Reports (impact factor ≥ 1.5). For each journal, five orig…
The gain
In a study of 800 evaluations of 50 medical abstracts under four fictional author identities, ChatGPT and Claude showed no consistent gender or geographic bias and scored with high reproducibility.
What this doesn’t fix
High agreement may be inflated by narrow scoring, and validity compared to human peer review remains untested.
Sources
1 on file- Journal of General Internal Medicinedoi:10.1007/s11606-026-10806-8
Cite this record
v3Truvace Impact Record TRV-2026-1274, v3: “Bias and Reliability of AI-Based Peer Review: A Comparative Study of ChatGPT and Claude Evaluating Scientific Abstracts.” Truvace, 2026-11-05. /record/TRV-2026-1274 (accessed on the day you copy it). sha256 89eef90b48bf6653…
Change history
3 versionsEvery change made to this record since it was first published, and why.
Voice pass: plainer wording and study attribution; no facts or figures changed.
Editorial pass: sharper headline; gain/problem restated as concrete outcomes; refiled trace to g_space (gain outcome, problem absent).
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-1274 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.
ace