Truace
Subscribe
48/100

Emerging evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Claude beat ChatGPT and Gemini on child paracetamol poisoning cases, but two made critical errors

Study title: Comparative performance and safety of three large language models in pediatric paracetamol poisoning management

Evidence type
Benchmark or lab evaluation
Sample
n = 90
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2870cite or check it

Study title: Comparative performance and safety of three large language models in pediatric paracetamol poisoning management

A scenario-based study tested ChatGPT 5.5, Claude Sonnet 4.6 and Gemini 3 Flash on 30 hypothetical pediatric paracetamol poisoning cases, yielding 90 responses. A blinded panel of two general pediatricians and one pediatric emergency specialist scored each one against a guideline-derived reference algorithm. Claude scored 24.50 out of 25 on the study's quality rubric, versus 21.00 for ChatGPT and 20.07 for Gemini.

Read the study abstract

Abstract Background Paracetamol overdose is a common reason for pediatric poisoning consultations and emergency department visits, and. Large language models (LLMs) are increasingly being explored for clinical decision support, but evidence on pediatric poisoning is scarce. We compared the performance and safety of three contemporary LLMs for managing pediatric paracetamol poisoning using guideline-based simulated scenarios. Methods In this cross-sectional, comparative, scenario-based study, 30 hypothetical pedi…

54Gain side

How the two sides scoreClose call

51Problem side

The gain

In a study of 30 simulated pediatric paracetamol poisoning cases, Claude Sonnet 4.6 scored 24.50 of 25 on expert-rated quality, made no critical safety errors and matched the reference guideline in all 30 responses.

The problem

In the same 30-scenario test of pediatric paracetamol poisoning, Gemini 3 Flash made critical safety errors in 3 of 30 responses and ChatGPT 5.5 in 1 of 30, including missed lab checks and risk of stopping N-acetylcysteine early.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Claude also scored 7.93 of 8 on the key-action checklist, against 7.13 for ChatGPT and 6.90 for Gemini, and 4.90 of 5 on the Global Quality Scale. It had no critical safety errors, while ChatGPT had one and Gemini had three. That error-rate difference was not statistically significant, and the authors say the test was underpowered.

The scenarios were hypothetical and the scores came from a single panel consensus using a rubric that has not been externally validated. The authors conclude chatbots may serve as adjunct decision support in pediatric toxicology but only under expert supervision with clinical verification.

Main points

  1. The study used 30 scenarios written by pediatricians, covering acute, delayed, unknown-dose, repeated, modified-release, malnutrition and co-ingestion cases.
  2. Every model got the same single-turn prompt in its public browser interface, with no follow-ups, plug-ins or browsing.
  3. Claude got every N-acetylcysteine indication right (30/30), versus 22/30 for ChatGPT and 24/30 for Gemini.
  4. Claude got the mg/kg dose right in 30/30 cases; ChatGPT managed 26/30 and Gemini 22/30.
  5. ChatGPT and Gemini did not differ significantly on total quality score.
  6. Claude's lead was widest in advanced decisions such as NAC stopping and acidosis, where it scored 25 in all six scenarios against a median of 18 for ChatGPT and 22 for Gemini.
  7. Concordance with the reference standard was 30/30 for Claude, 26/30 for ChatGPT and 27/30 for Gemini.

The full story

The scenarios were developed from the Australia and New Zealand paracetamol poisoning guidelines plus pediatric emergency guidance from the Royal Children's Hospital Melbourne and Queensland Paediatric Emergency Care. Responses were anonymized and randomized before a three-clinician panel scored them by consensus. Scoring used a five-domain rubric, an eight-item checklist, the Global Quality Scale and a critical-error flag.

Critical errors were defined as omissions that could affect safety, such as missing an NAC indication or omitting liver tests in delayed presentation. ChatGPT's one error was missing lab evaluation in repeated supratherapeutic ingestion. Gemini's three involved inadequate evaluation of an unknown-amount ingestion, missed lab evaluation in repeated ingestion and risk of premature NAC termination.

By category, Claude led in acute, atypical, special-situation and advanced scenarios. Gemini's scores varied most in unknown-time or delayed-presentation cases. Checklist results hit a ceiling for Claude, limiting how well the test separated the models.

What this doesn’t fix

Responses were scored by a single three-clinician consensus without inter-rater reliability, using an unvalidated rubric, on hypothetical cases with specific model versions on one date; critical errors were too rare (0–3 of 30) for the comparison to be statistically conclusive.

Sources

1 on file
  1. BMC Pediatricsdoi:10.1186/s12887-026-07949-2
    Peer-reviewed2026-11-11

Talk it over

Three questions to start on, two more if you have timeClaude Sonnet 4.6 made no critical safety errors in 30 scenarios, while Gemini 3 Flash made 3 and ChatGPT 5.5 made 1. How would you weigh that gap against the fact that the test was underpowered?

Start with a question

Claude beat ChatGPT and Gemini on child paracetamol poisoning cases, but two made critical errorshttps://truvace.com/item/comparative-performance-and-safety-of-three-large-language-models-in-pediatric-p

  1. Claude Sonnet 4.6 made no critical safety errors in 30 scenarios, while Gemini 3 Flash made 3 and ChatGPT 5.5 made 1. How would you weigh that gap against the fact that the test was underpowered?

    Answer in the thread

  2. A panel of two general pediatricians and one pediatric emergency specialist scored the responses by consensus, using a rubric not externally validated. How much should that scoring setup shape what we take from the 24.50 versus 21.00 versus 20.07 results?

    Answer in the thread

  3. If you were a pediatrician on that panel and saw Gemini's response risk stopping N-acetylcysteine early, what would you want to check before trusting any chatbot answer in a real poisoning case?

    Answer in the thread
If you have more time

  • The scenarios came from Australia and New Zealand guidelines and Royal Children's Hospital Melbourne guidance. How might results differ with cases built from guidelines used elsewhere, and where could you look to find out?

    Answer in the thread

  • Each model got one single-turn prompt in its public browser interface. What would you want to test if a clinician could ask follow-up questions, and what result would change your view?

    Answer in the thread

What readers say

More from Health