Truace
Subscribe

LLM performance and evaluation on clinically oriented anatomy multiple-choice questions for medical education

Source article: Benchmarking Large Language Models (LLMs) Against Medical Students Using Psychometrically Evaluated Authentic Clinically Oriented Anatomy Multiple-Choice Questions (MCQs) at the Apply and Analyze Levels of Bloom's Taxonomy

Large language models (LLMs) are increasingly used by medical students as study aids, yet evidence regarding their performance on higher-order, clinically oriented anatomy questions relative to that of students-particularly at the Apply and Analyze levels of Bloom's Taxonomy-remains limited. This study aimed to compare the accuracy of five LLMs (ChatGPT-5, Gemini 2.5 Pro, Claude, Microsoft Copilot, and DeepSeek R1) with that of medical students on authentic, psychometrically evaluated anatomy multiple-choice que…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1428· cite & verify
Benchmarking Large Language Models (LLMs) Against Medical Students Using Psychometrically Evaluated Authentic Clinically Oriented Anatomy Multiple-Choice Questions (MCQs) at the Apply and Analyze Levels of Bloom's Taxonomy
"Chatbots linked" by Ashhstarr is licensed under CC BY-SA 4.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-sa/4.0/.
Trace impact readingPositive state
P 64The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 69The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers at Bursa Uludag University Faculty of Medicine gave 20 clinically oriented anatomy multiple-choice questions to five LLMs: ChatGPT-5, Gemini 2.5 Pro, Claude, Microsoft Copilot and DeepSeek R1. Each model answered every question correctly, while students from the 2025-2026 academic year averaged 67.95%. Student accuracy dropped from Apply-level to Analyze-level questions, but the LLMs stayed at ceiling.

The results suggest current LLMs could serve as supplementary anatomy study tools, since they were unaffected by difficulty features that separated student performance. Yet the models' high ratings of each other's output raise doubts about using LLMs to grade LLMs. Because the test was small and every model scored perfectly, the study could not distinguish the models by accuracy.

Main points

  1. Students scored 74.79% on Apply-level items and 61.13% on Analyze-level items, a significant decline (p = 0.02).
  2. Analyze-level questions discriminated between students more strongly than Apply-level ones (0.446 vs. 0.248; p < 0.001).
  3. On rubric scoring, Gemini 2.5 Pro and Claude scored 10.0/10 for anatomical accuracy and reasoning, while Microsoft Copilot scored lowest at 8.0/10 per human observers.
  4. The 20 questions were selected from a pool of 397 and each was presented once to every model using a standardized prompt.

The gain

All five tested LLMs answered every one of 20 Apply- and Analyze-level anatomy questions correctly, while students averaged about two-thirds accuracy on the same items.

The problem

When LLMs rated each other's anatomy answers, they scored them as good as or better than human observers did, never lower, which undermines the reliability of LLM-based peer assessment.

The rundown

The study was a retrospective, cross-sectional analysis. Ten Apply-level and ten Analyze-level questions were drawn from a 397-question pool and analyzed alongside anonymized, aggregated student examination records. Accuracy was compared with Fisher's exact test, and item difficulty and discrimination were compared with independent-samples t-tests.

Response quality was scored using a six-domain anatomy rubric by two human observers and by peer LLMs in a cross-evaluation design. All models scored maximally on correct answer selection, Bloom's taxonomy appropriateness, scientific consistency and absence of hallucination. Differences appeared only in anatomical accuracy and anatomical reasoning.

Because LLM accuracy had zero variance, inter-model statistical comparisons were not applicable. The authors conclude the findings support LLMs as supplementary tools while stressing the continued need for expert oversight.

What this doesn’t fix

The study used only 20 questions from a single institution, each given once to the models, and perfect LLM scores left zero variance, so the models could not be statistically compared.

Sources

  1. Peer-reviewedClinical Anatomy2026-10-01

The debate