AI chatbots left out critical risks in jaw surgery consent forms, a study of 132 outputs found

Critical omissions were common for two of the tools. Fifteen of 33 ChatGPT outputs and 17 of 33 Gemini outputs missed at least one critical item, versus one of 33 each for DeepSeek and Grok. Of 34 affected outputs, 28 met general-consent criteria and six met procedure-specific criteria, so the gaps often involved procedure-specific content.

The authors say aggregate coverage did not establish completeness or clinical safety. Six outputs carried incorrect or unsupported-statement flags and two carried over-reassurance flags. Patient comprehension and clinical usefulness were not tested, so the results do not show how the documents would perform with real patients.

The team used a fixed corpus of 132 outputs: four free consumer interfaces, 11 orthognathic and craniofacial procedures and three new-chat repetitions each. Three oral and maxillofacial surgeons scored every output independently with a locked checklist, and final results used item-level two-of-three majorities.

Coverage comparisons used 11 paired procedure blocks, and critical omissions were analysed with a procedure-random-intercept logistic model. Agreement between surgeons was high at item level (Gwet AC1 0.892, 95% CI 0.885-0.899) but lower for derived critical-status judgments (AC1 0.583, 0.477-0.689).

Of 34 outputs with critical omissions, 28 met general-consent or task criteria and six met procedure-specific criteria. Excluding the one format failure left the coverage comparisons unchanged. Readability comparisons were descriptive only, with mean grade levels between 10.22 and 13.89.