aceModerate evidence
More
13 AI chatbots scored 73% to 97% on hematology decision questions, a test of 30 cases
Study title: An Analysis of Large Language Model Performance on Hematology Key Feature Questions
- Evidence type
- Benchmark or lab evaluation
- Sample
- n = 390
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2866cite or check it
Study title: An Analysis of Large Language Model Performance on Hematology Key Feature Questions
Researchers tested 13 large language models in June 2026 on 30 new hematology key feature questions, which assess clinical decision-making at critical points without offering answer choices. Across 390 responses, exact accuracy ranged from 73.3% to 96.7%.
Read the study abstract
Large language models (LLMs) demonstrate strong performance on multiple-choice questions but their ability to independently generate clinical decisions without pre-defined answer choices remains less established. We therefore evaluated LLM performance on post-graduate key feature questions (KFQs), which are designed to assess clinical decision-making at critical points. Thirteen LLMs were assessed in June 2026 using 30 novel hematology KFQs developed and reviewed by two hematologists with training in medical edu…
53Gain side
How the two sides scoreClose call
53Problem side
The gain
In a test of 13 AI models on 30 new postgraduate hematology key feature questions, exact accuracy ranged from 73.3% to 96.7%, with Gemini 3 Flash answering 29 of 30 correctly.
The problem
In the same 30-question hematology test of 13 AI models, 71 of 390 responses were wrong, and 49.3% of those errors were knowledge errors and 45.1% were decision errors.
A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.
What the scores mean
G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.
P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.
In brief
Gemini 3 Flash scored highest at 29 of 30 (96.7%). Gemini 3 Flash Force Thinking and OpenEvidence each got 27 of 30 (90.0%), while DeepSeek V3 scored lowest at 22 of 30 (73.3%). Differences between models were not statistically significant.
Of 71 wrong answers, 35 were knowledge errors, 32 were decision errors and four were comprehension errors. The authors say this supports checking factual accuracy and clinical judgment alongside overall scores.
Main points
- Thirteen LLMs were assessed in June 2026 using 30 novel hematology key feature questions written and reviewed by two hematologists trained in medical education.
- Each model assessed every case independently, with context reset between cases, producing 390 responses.
- Exact accuracy ranged from 73.3% to 96.7%; Gemini 3 Flash led with 29 of 30 correct (95% CI, 82.8-99.9%).
- Gemini 3 Flash Force Thinking and OpenEvidence both scored 27 of 30 (90.0%); DeepSeek V3 scored 22 of 30 (73.3%).
- Model differences were not significant for exact correctness (Cochran's Q = 20.00; p = 0.067) or partial credit (Friedman's χ² = 17.95; p = 0.117).
- Among 71 incorrect responses, 49.3% were knowledge errors, 45.1% decision errors and 5.6% comprehension errors.
The full story
The study targets a gap: LLMs perform well on multiple-choice questions, but their ability to reach clinical decisions without pre-defined answer choices is less established. Key feature questions were used because they test decisions at critical points in a case.
Responses were scored for exact correctness and partial credit and compared using mixed-effects regression, Cochran's Q and Friedman's test. None showed statistically significant differences between the models.
Incorrect answers were sorted into knowledge, decision and comprehension errors. Knowledge and decision errors were close to evenly split, while comprehension errors were rare. The authors call for research on whether learners can spot model errors and whether LLM help improves hematology decision-making.
What this doesn’t fix
The question bank was limited to 30 questions, which the authors say precludes conclusions about equivalent performance between models.
Sources
1 on file- Journal of Cancer Educationdoi:10.1007/s13187-026-03025-6
Talk it over
Gemini 3 Flash got 29 of 30 right, yet 71 of 390 responses across the 13 models were wrong. How would you weigh the high scores against the mix of knowledge errors and decision errors?
13 AI chatbots scored 73% to 97% on hematology decision questions, a test of 30 caseshttps://truvace.com/item/an-analysis-of-large-language-model-performance-on-hematology-key-feature-questi
Gemini 3 Flash got 29 of 30 right, yet 71 of 390 responses across the 13 models were wrong. How would you weigh the high scores against the mix of knowledge errors and decision errors?
Answer in the threadThe 30 questions were written and reviewed by two hematologists trained in medical education, and the model differences were not statistically significant. What can a test of this size tell us about which model is better?
Answer in the threadWhat result would make you trust a model to help with hematology decisions: fewer decision errors, a larger question bank, or a test with learners who check the answers?
Answer in the thread
If you have more time
Look up how key feature questions differ from multiple-choice questions. How might a model that does well on one type do on the other?
Answer in the threadThe authors want research on whether learners can spot model errors. How would you design a small study to test that, and who would you recruit?
Answer in the thread
What readers say