Truace
Subscribe

AI chatbots gave mostly safe answers to jaw surgery aftercare questions, but one missed a risk

Study title: Large language model responses to postoperative questions in oral and maxillofacial surgery: a comparative evaluation

Researchers put 30 routine aftercare questions from oral and maxillofacial surgery to three chatbots: ChatGPT-5.4 Thinking, DeepSeek-V3.2 and Gemini 3. Three surgeons, blinded to which model wrote each answer, scored quality, accuracy, completeness and safety, and the team also measured readability and response time.

Read the study abstract

Abstract Background Large language models (LLMs) are increasingly used as patient-facing informational tools in dentistry, but their suitability for postoperative communication in oral and maxillofacial surgery (OMFS) remains uncertain. This study compared ChatGPT-5.4 Thinking, DeepSeek-V3.2, and Gemini 3 (Thinking mode) in answering routine postoperative OMFS questions. Methods Thirty standardised open-ended postoperative questions were submitted once to each model through public web interfaces using the same n…

By the Truvace desk1 min read1 sourceEvidence: Emerging · 48/100 — Benchmark or lab evaluation · n = 90 · 1 independent sourcePeer-reviewedTRV-2026-2354· cite & verify
Large language model responses to postoperative questions in oral and maxillofacial surgery: a comparative evaluation
Odontogenic sinusitis by Ryanlittltemd. CC BY-SA 4.0 · https://creativecommons.org/licenses/by-sa/4.0
Trace impact readingNegative state
P 52The P score combines the specificity and measured human impact of the grounded problem claim with the strength of this Trace’s cited sources.

Both sides are scored from claims and sources, not community votes.

G 47The G score combines the specificity and measured human impact of the grounded gain claim with the strength of this Trace’s cited sources.

In brief

Gemini scored highest on expert-rated quality, but accuracy differences between models did not hold after the ratings were reassessed, and completeness did not differ. One answer on resuming clopidogrel gave false reassurance. The test used simulated single-turn English questions, so it says little about whether real patients understand the answers or are kept safe.

Main points

  1. Mean quality scores were 4.82 for Gemini 3, 4.53 for DeepSeek-V3.2 and 4.08 for ChatGPT-5.4 Thinking.
  2. 89 of 90 responses were rated safe; one Gemini answer on clopidogrel was a missed red flag.
  3. Original accuracy and completeness ratings showed limited rater agreement, and accuracy differences disappeared after reassessment.
  4. Gemini kept its quality advantage after adjusting for word count, but its answers were more linguistically complex and slower.

The gain

Expert surgeons rated the chatbot answers highly for quality, led by Gemini 3 at 4.82 out of 5, and 89 of 90 responses were classified as safe.

The problem

One Gemini response on clopidogrel resumption was classed as a missed red flag giving false reassurance, and factual accuracy ratings showed limited agreement between raters.

The rundown

Thirty standardised open-ended questions were submitted once to each model through public web interfaces using the same neutral prompt. Three surgeons rated overall quality on an adapted 5-point Global Quality Score, with excellent inter-rater reliability.

Accuracy, completeness and clinical safety were judged against a question-specific reference answer and safety framework. Accuracy differed in the original ratings (p = 0.014) but not after reassessment (p = 0.115).

DeepSeek-V3.2 had the shortest observed latency at 10.49 seconds and Gemini the longest at 31.46 seconds, though the authors say this does not reflect intrinsic model speed. The authors conclude that chatbot-generated postoperative information should remain under professional oversight.

What this doesn’t fix

The findings are limited to simulated, single-turn, English-language first responses and do not establish patient comprehension or real-world safety.

Sources

  1. Peer-reviewedBMC Oral Health2026-11-03

The debate