aceModerate evidence
Published recordHealthThe Trace
More
An Analysis of Large Language Model Performance on Hematology Key Feature Questions
- Truvace record
- TRV-2026-2866raw data (JSON)
- Version
- v1Published
- First published
- Evidence type
- Benchmark or lab evaluation
- Sources
- 1 sourcePeer-reviewed
- sha256
- c601aac7ca66…check it
Large language models (LLMs) demonstrate strong performance on multiple-choice questions but their ability to independently generate clinical decisions without pre-defined answer choices remains less established. We therefore evaluated LLM performance on post-graduate key feature questions (KFQs), which are designed to assess clinical decision-making at critical points. Thirteen LLMs were assessed in June 2026 using 30 novel hematology KFQs developed and reviewed by two hematologists with training in medical edu…
53Gain side
How the two sides scoreClose call
53Problem side
The gain
In a test of 13 AI models on 30 new postgraduate hematology key feature questions, exact accuracy ranged from 73.3% to 96.7%, with Gemini 3 Flash answering 29 of 30 correctly.
The problem
In the same 30-question hematology test of 13 AI models, 71 of 390 responses were wrong, and 49.3% of those errors were knowledge errors and 45.1% were decision errors.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
What this doesn’t fix
The question bank was limited to 30 questions, which the authors say precludes conclusions about equivalent performance between models.
Sources
1 on file- Journal of Cancer Educationdoi:10.1007/s13187-026-03025-6
Cite this record
v1Truvace Impact Record TRV-2026-2866, v1: “An Analysis of Large Language Model Performance on Hematology Key Feature Questions.” Truvace, 2026-11-12. /record/TRV-2026-2866 (accessed on the day you copy it). sha256 c601aac7ca66b723…
Change history
1 versionEvery change made to this record since it was first published, and why. Nothing has changed since it went up.
Certified into the record
Check this record hasn’t been altered
How to check it yourself, without trusting this page
Download any version from /api/record/TRV-2026-2866 and hash its canonical field yourself, for example with shasum -a 256. You should get its content_hash. Each version’s text also ends with prev:and the hash of the version before it (version 1 points to 64 zeros). If anyone had changed a single character of any version after it was saved, those hashes wouldn’t line up.