Truace
Subscribe

Chatbot answers to public questions about altered consciousness

Source article: Evaluating Generative Artificial Intelligence Chatbots in Answering Public Questions About Altered Consciousness: A Comparative Study of Safety, Accuracy, Reliability, and Readability

Introduction People increasingly use generative artificial intelligence chatbots to interpret health concerns before contacting clinicians. Although these systems are not formal autonomous emergency department triage tools, their responses may influence whether and when users seek professional care. Altered consciousness is especially high-risk because delayed or ambiguous escalation can postpone time-sensitive evaluation. This study evaluated the safety and quality of chatbot answers from an emergency nursing p…

By the Truvace desk2 min read1 sourcePeer-reviewedTRV-2026-1746· cite & verify
Evaluating Generative Artificial Intelligence Chatbots in Answering Public Questions About Altered Consciousness: A Comparative Study of Safety, Accuracy, Reliability, and Readability
Construction site tour of the Skilled Nursing Residence, in Edgartown, on the island of Martha's Vineyard, in Dukes County, Maryland on August 5, 2024 - 42 by USDAgov. Public domain
Trace impact readingContested
P 70The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 69The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Researchers compared ChatGPT, Gemini, Claude, DeepSeek and Doubao on 66 public-facing English questions spanning 11 emergency consultation domains. Each question was submitted once per chatbot in May 2026, and five experienced nurses rated the 330 resulting responses for safety, accuracy, empathy, quality, transparency and readability.

Because people use chatbots to interpret symptoms before contacting clinicians, advice on altered consciousness may affect how quickly someone seeks urgent care. The study found mostly safe answers but also unsafe ones from every model. It was a single-run snapshot, so it cannot show how users act on the advice or whether results hold over time.

Main points

  1. Interrater agreement was good to excellent, with a Fleiss kappa for safety of 0.848.
  2. Safe-response rates ranged from 89.4% (Doubao) to 95.5% (ChatGPT), with no significant overall difference.
  3. ChatGPT had the highest mean accuracy and the top DISCERN and JAMA benchmark scores.
  4. Claude and Gemini were rated most empathic, and DeepSeek produced the most readable text across most formulas.
  5. Authors recommend emergency nurses help shape escalation-first, plain-language safety templates.

The gain

Across 330 chatbot responses, nurse-rated safe-response rates were high, from 89.4% to 95.5%, suggesting chatbots can support public education about altered consciousness.

The problem

Every tested chatbot produced at least one potentially unsafe answer, with omissions, unsafe sequencing and urgency-softening making them unsuitable for autonomous triage.

The rundown

The study used a 130-item candidate pool screened down to 66 English questions covering 11 emergency consultation domains. Each was submitted once to each chatbot in a fresh conversation between May 1 and May 30, 2026, yielding 330 responses.

Five clinically experienced nurses independently scored the responses. Intraclass correlation coefficients for nonbinary nurse-rated outcomes ranged from 0.836 to 0.878, indicating consistent ratings.

Performance varied by dimension: ChatGPT led on accuracy and information-quality instruments, Claude and Gemini on empathy, and DeepSeek on readability. The authors state these tools are not formal autonomous emergency department triage systems.

What this doesn’t fix

Each question was submitted only once per chatbot, and the paper itself concludes the tools are unsuitable for autonomous triage because every model produced at least one potentially unsafe answer.

Sources

  1. Peer-reviewedJournal of Emergency Nursing2026-10-07

The debate