48/100

Emerging evidence

Cite
More
EmailLinkedInX

AI chatbots misread dental scans: a 90-patient study found Class III bite problems missed

Study title: Large language models for orthodontic classification using intraoral digital models: a diagnostic accuracy and reproducibility study

Evidence type
Benchmark or lab evaluation
Sample
n = 90
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Large language models for orthodontic classification using intraoral digital models: a diagnostic accuracy and reproducibility study

A retrospective diagnostic accuracy study tested Claude Opus 4.7, GPT-5.5 and Gemini 3 Flash on archived iTero intraoral scans from 90 patients, 30 in each Angle class. Overall accuracy ranged from 29.2% (Gemini 3 Flash, arch form) to 64.0% (GPT-5.5, arch form), and kappa agreement with the expert reference never exceeded 0.299.

Read the study abstract

Abstract Background Multimodal artificial intelligence (AI) systems built on large language models are consulted by patients and clinicians, but their performance on visual orthodontic classification is not well characterized. This study evaluated three such systems for agreement with an expert orthodontist reference standard, category-specific diagnostic performance and four-week reproducibility, using intraoral digital scan exports. Methods In this retrospective diagnostic accuracy study, 90 patients were sele…

Large language models for orthodontic classification using intraoral digital models: a diagnostic accuracy and reproducibility study
"A Navy dentist exames a patient at sea" by Official U.S. Navy Page, public domain.

The problem

In a retrospective study of 90 patients' intraoral scans, Claude Opus 4.7, GPT-5.5 and Gemini 3 Flash agreed poorly with orthodontists (kappa up to 0.299) and detected only 10.0–13.3% of Class III cases versus 60.0–80.0% of Class I.

In brief

The chatbots were weakest on Class III malocclusion, where sensitivity was 10.0–13.3% against 60.0–80.0% for Class I. The authors conclude that overall accuracy hid this category-level weakness and that the systems are not suitable for standalone orthodontic diagnosis under the conditions tested.

Results were also unstable over time. When 20 patients were reassessed after four weeks, sagittal accuracy fell from 55.0–60.0% to 5.0–35.0%, and most test-retest kappa confidence intervals included zero.

Main points

  1. The study used 90 archived patient records, with 30 patients in each Angle class.
  2. Five standardized two-dimensional views from each iTero scan were given to each chatbot with a single zero-shot prompt.
  3. Two orthodontists classified each case from the complete scan record with metric measurements; their agreed categories formed the reference standard.
  4. Macro-average recall ranged from 38.7% (Gemini 3 Flash, arch form) to 60.9% (GPT-5.5, vertical relationship).
  5. Class III sensitivity was 10.0–13.3%, compared with 60.0–80.0% for Class I.
  6. Test-retest kappa ranged from −0.238 (Gemini 3 Flash, sagittal classification) to 0.433 (GPT-5.5, vertical relationship).
  7. Classifying the same five images, the orthodontists agreed with the reference standard in 96.6–100% of cases.

The full story

Researchers assessed four parameters: sagittal classification, vertical relationship, crowding severity and arch form. Accuracy, class-specific sensitivity and specificity, macro-average recall and Cohen's kappa were calculated with 95% confidence intervals.

To check whether the images themselves were the problem, the orthodontists also classified all cases from the same five images. They agreed with the reference standard in 96.6–100% of cases, and each reproduced at least 19 of 20 classifications for each parameter at retest.

The chatbots did not hold steady over time. In the 20-patient retest after four weeks, sagittal accuracy fell from 55.0–60.0% to 5.0–35.0%, and most test-retest kappa intervals included zero. The authors say reproducibility must be evaluated separately from accuracy.

Sources

1 on file
  1. BMC Oral Healthdoi:10.1186/s12903-026-10465-5
    Peer-reviewed2026-11-14

Talk it over

Three questions to start on, two more if you have time
The chatbots detected only 10.0% to 13.3% of Class III cases but 60.0% to 80.0% of Class I. When judging a tool like this, how would you weigh an overall accuracy figure against the result for each class?

Start with a question

AI chatbots misread dental scans: a 90-patient study found Class III bite problems missedhttps://truvace.com/item/large-language-models-for-orthodontic-classification-using-intraoral-digital-mod

  1. The chatbots detected only 10.0% to 13.3% of Class III cases but 60.0% to 80.0% of Class I. When judging a tool like this, how would you weigh an overall accuracy figure against the result for each class?

    Answer in the thread
  2. Researchers gave each chatbot five two-dimensional views from the iTero scans with one zero-shot prompt, while the orthodontists used the complete scan record. What can that setup tell us about how the chatbots would do with real patients?

    Answer in the thread
  3. When 20 patients were reassessed after four weeks, sagittal accuracy fell from 55.0% to 60.0% down to 5.0% to 35.0%. What result on repeat testing would make you trust a chatbot's orthodontic classification enough to use it?

    Answer in the thread
If you have more time
  • Which other prompts, image sets or newer chatbot versions would you want tested to decide whether the Class III weakness is built in or a product of this setup?

    Answer in the thread
  • In a dental clinic you know, where might a chatbot classification be used as a first look, and who would check it against the orthodontists' own reading?

    Answer in the thread

What readers say