Truace
Subscribe
52/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

AI chatbots scored 65% on radiology image questions versus 95% on text, a test of seven found

Study title: Enhancing medical education with AI: A comparative study of various large language models in radiology

Evidence type
Benchmark or lab evaluation
Sample
n = 120
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2464cite or check it

Study title: Enhancing medical education with AI: A comparative study of various large language models in radiology

A study at King Abdulaziz University's Faculty of Medicine in Rabigh, Saudi Arabia, tested seven AI chatbots on 120 newly written radiology multiple-choice questions for fourth-year medical students. Across all models the mean accuracy was 79.9% (671 of 840 answers correct). Average accuracy fell from 95% on 60 text questions to 65% on 60 image questions.

Read the study abstract

Objective To compare how various Large Language Models (LLMs) perform when solving text- and image-based medical student-level radiology multiple-choice questions (MCQs). Methods We evaluated the accuracy of seven LLMs (i.e., ChatGPT-4o, ChatGPT-5, ChatGPT-5 Plus, DeepSeek-V3.1-Terminus, Microsoft Copilot, Claude 4, and Gemini 2.5 Flash) in solving 60 text-based and 60 image-based MCQs. We compared the seven LLMs' accuracy across question types, domains, and topics. Results The mean accuracy across the seven eva…

The problem

In a test of seven AI chatbots on 120 medical-student radiology questions, average accuracy was 65% on image-based questions versus 95% on text-based ones, so image answers need faculty checking.

In brief

Claude 4 scored highest at 87% (104 of 120), while Gemini 2.5 Flash and Microsoft Copilot scored lowest at 75% each. Every model dropped 20 to 38 percentage points when moving from text to images. The authors conclude the chatbots should serve as add-ons to teaching, with faculty supervision, rather than replace it.

The differences between individual models were weak. Claude's lead over Gemini and Copilot did not hold up after statistical correction for multiple comparisons, so the consistent text-to-image drop is the firmer finding. Free models, DeepSeek-V3.1-Terminus and ChatGPT-4o at 82% each, slightly beat the paid ChatGPT-5 Plus at 80%.

Main points

  1. Seven chatbots were tested: ChatGPT-4o, ChatGPT-5, ChatGPT-5 Plus, DeepSeek-V3.1-Terminus, Microsoft Copilot, Claude 4 and Gemini 2.5 Flash, each answering 60 text and 60 image questions.
  2. Claude 4 scored 98% (59 of 60) on text questions and 75% (45 of 60) on image questions; Gemini scored 58% and Copilot 57% on image questions.
  3. Knowledge-based questions averaged 83% accuracy versus 77% for skills-based questions.
  4. By topic, accuracy ranged from 70% for musculoskeletal imaging to 100% for radiotherapy, but some topics had only three or four questions.
  5. Free DeepSeek-V3.1-Terminus and ChatGPT-4o (82% each) slightly outscored paid ChatGPT-5 Plus (80%).
  6. In the statistical model, question type strongly predicted correctness (p < 0.001), while the model used did not (p = 0.361).
  7. The authors recommend faculty supervision, teaching students to critically appraise AI output, and treating the tools as aids for revision and self-testing.

The full story

Two radiology assistant professors wrote the questions to match fourth-year undergraduate exams, using institutional teaching materials, and sorted them by type, domain and topic. Image questions used anonymized X-rays, CT, ultrasound, fluoroscopy and nuclear scans. All seven models were tested between August 23 and September 2, 2025, through public consumer interfaces with a single zero-shot prompt and one attempt per question.

Text questions were submitted in batches of 10, while image questions were submitted one at a time. Analysis used generalized estimating equations, Cochran's Q test and Bonferroni-corrected pairwise McNemar tests. Cochran's Q showed an overall difference among models (p = 0.035), but no pairwise gap survived correction (Claude versus Gemini, p = 0.096; versus Copilot, p = 0.180).

The authors say the study cannot separate failure to read images from other factors. Image questions skewed toward skills (66.7%) while text questions were more evenly split. The authors also say some image answers may have been deduced from the question text, so the 30-point gap may overstate the image-reading deficit.

They describe the 79.9% overall figure as driven by strong text performance and not a measure of image-reading skill. They note that confidently wrong answers are a documented risk, though this study did not measure how often they occurred, and call for careful governance of how students use these tools.

Talk it overFour questions for a class, a team or the dinner table

  1. Who carries this cost, and what would have to change to prevent it?

    The problem In a test of seven AI chatbots on 120 medical-student radiology questions, average accuracy was 65% on image-based questions versus 95% on text-based ones, so image answers need faculty checking.

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a benchmark or lab test with a sample of 120, from PLOS One.

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The 120 questions were written at a single institution by two faculty members without pilot testing, there was no human comparison group, and text questions were batched while image questions were not, so the gap may overstate image-reading weakness.

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

The questions are the same on every record. The details under them are taken from this record as written.

Sources

1 on file
  1. PLOS Onedoi:10.1371/journal.pone.0360928
    Peer-reviewed2026-11-06

The debate

More from Health