Truace
Subscribe
70/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

A review of 12 studies shows AI chatbots got 69.6% of nursing exam questions right

Study title: Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis

Evidence type
Systematic review or meta-analysis
Sample
n = 13,870
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2792cite or check it

Study title: Performance of large language models on nursing licensure examinations: A systematic review and meta-analysis

A systematic review and meta-analysis in Nurse Education Today pooled 12 studies in which AI chatbots answered multiple-choice questions from nursing licensure exams. The studies covered 13,870 questions, five exam systems and eight models. Overall accuracy was 69.6% (95% CI 65.6–73.6%), with a prediction interval of 43.8% to 95.3%.

Read the study abstract

OBJECTIVES: This systematic review and meta-analysis assessed the performance of large language models (LLMs) in nursing licensure examinations. Despite the increasing use of LLMs in healthcare education, their capabilities in nursing licensure examinations remain uncertain. This study provides evidence on the accuracy and limitations of LLMs to help guide their integration into nursing education and licensure. DESIGN: The systematic review and meta-analysis adhered to PRISMA 2020 guidelines. DATA SOURCES: PubMe…

57Gain side

How the two sides scoreClose call

61Problem side

The gain

In a meta-analysis of 12 studies, GPT-4 answered 77.2% of nursing licensure questions correctly versus 60.4% for GPT-3.5, and a domain-customised model reached 93.6%.

The problem

A systematic review of 12 studies covering 13,870 nursing licensure questions found pooled chatbot accuracy of 69.6% with wide variation (I² = 98%), weak ethics and psychosocial results, and a recommendation against unmoderated high-stakes use.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Newer and customised models did much better than early ones. GPT-4 reached 77.2% against 60.4% for GPT-3.5, Qwen 2.5 reached 88.9%, and a task-specific Custom GPT reached 93.6%. Google Bard scored lowest at 53.3%. The authors conclude the tools suit low-stakes uses such as formative assessment but not unmoderated licensure decisions.

The pooled figure hides large gaps between studies (I² = 98%). Chatbots did well in general medicine and pharmacology but poorly in ethics and psychosocial integrity, and accuracy fell when NCLEX-RN items were translated. Nine of the 12 studies came from East Asia, so results for Western exams rest on less evidence.

Main points

  1. The review searched PubMed, CINAHL, PsycINFO, EMCARE and ERIC from April to June 2025 and included 12 studies, which assessed 13,870 multiple-choice questions.
  2. Pooled accuracy was 69.6% (95% CI 65.6–73.6%) with a wide prediction interval of 43.8–95.3% and very high heterogeneity (I² = 98%).
  3. GPT-4 scored 77.2% against 60.4% for GPT-3.5; GPT-4o scored 80.7%, Qwen 2.5 88.9%, a Custom GPT 93.6%, and Google Bard 53.3%.
  4. Chatbots were strongest in general medicine (88.75%) and weakest in psychosocial integrity, where one included study reported 13.3%.
  5. Translating NCLEX-RN items into Chinese lowered accuracy significantly (p = 0.03), while translated NNLE items showed no such drop (p = 0.92).
  6. Image-based questions were a weak spot: one included study reported 56.3% accuracy on visual items.
  7. Nine of the 12 studies came from East Asia (China, Japan, Taiwan), and temperature settings were reported in only two studies.

What the review found when it combined the studies

These are the review’s own pooled results, copied from its abstract. Truvace did not recalculate them.

  • Accuracy of large language models on multiple-choice nursing licensure exam questions

    Accuracy 69.6% (95% CI 65.6% to 73.6%)

    search to April to June 2025

    The review’s own words
    Pooled accuracy was 69.6% (95% CI: 65.6-73.6%) with substantial heterogeneity (I2 = 98%).

Code checks every number above against the review’s text before it is shown. The short labels are an AI model’s reading of that text, so the review’s own sentence sits under each one. A pooled figure is only as strong as the studies behind it. How Truvace uses pooled results

The full story

The authors screened 117 records and included 12 studies after excluding 27 at eligibility assessment. The studies tested models on exams from the United States (NCLEX-RN/PN), China (NNLE, CNNLE), Japan (JNNE), Taiwan (RNLE), Norway and Spain. GPT-3.5 appeared in eight studies and GPT-4 in four. Chinese models such as Qwen-2.5, ERNIE and SPARK appeared only in a single seven-model comparison.

Against national pass marks, results were mixed. GPT-4 scored 92.4% on NCLEX practice questions in Spain against a 77% benchmark, 80.75% on the Taiwan exam and 79.7% on the Japanese exam, while GPT-3.5 scored 59.9% there and missed the Japanese standard. In China, ChatGPT failed to reach the 60% threshold across five years of exams.

Complex formats lowered scores. One study found significant effects for clinical vignettes (p = 0.007) and complex multiple-choice items (p = 0.049), and another reported scenario-based accuracy from 51.7% to 80.0%. Even so, the pooled analysis found no significant differences by question difficulty level (p = 0.90) or by question format (p = 0.96).

The authors recommend regulators separate educational from high-stakes uses, avoid unmoderated use in licensure, and develop nursing-specific and culturally adapted models. They also call for standardised reporting, error analysis, comparisons with human test-takers and more research on Western exams.

What this doesn’t fix

The review included only English-language studies, relied on author-reported outcomes, and found that key settings such as temperature, prompting strategy and scoring were often unreported, which limits what the pooled figures can show.

Sources

1 on file
  1. Nurse Education Todaydoi:10.1016/j.nedt.2026.107154
    Peer-reviewed2026-05-12

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

A review of 12 studies shows AI chatbots got 69.6% of nursing exam questions righthttps://truvace.com/item/performance-of-large-language-models-on-nursing-licensure-examinations-a-systema

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a meta-analysis of 12 studies, GPT-4 answered 77.2% of nursing licensure questions correctly versus 60.4% for GPT-3.5, and a domain-customised model reached 93.6%.

    The problem A systematic review of 12 studies covering 13,870 nursing licensure questions found pooled chatbot accuracy of 69.6% with wide variation (I² = 98%), weak ethics and psychosocial results, and a recommendation against unmoderated high-stakes use.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a systematic review or meta-analysis with a sample of 13,870, from Nurse Education Today.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The review included only English-language studies, relied on author-reported outcomes, and found that key settings such as temperature, prompting strategy and scoring were often unreported, which limits what the pooled figures can show.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Education