Emerging evidence
More
AI chatbots misread dental scans: a 90-patient study found Class III bite problems missed
Study title: Large language models for orthodontic classification using intraoral digital models: a diagnostic accuracy and reproducibility study
- Evidence type
- Benchmark or lab evaluation
- Sample
- n = 90
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2967cite or check it
Study title: Large language models for orthodontic classification using intraoral digital models: a diagnostic accuracy and reproducibility study
A retrospective diagnostic accuracy study tested Claude Opus 4.7, GPT-5.5 and Gemini 3 Flash on archived iTero intraoral scans from 90 patients, 30 in each Angle class. Overall accuracy ranged from 29.2% (Gemini 3 Flash, arch form) to 64.0% (GPT-5.5, arch form), and kappa agreement with the expert reference never exceeded 0.299.
Read the study abstract
Abstract Background Multimodal artificial intelligence (AI) systems built on large language models are consulted by patients and clinicians, but their performance on visual orthodontic classification is not well characterized. This study evaluated three such systems for agreement with an expert orthodontist reference standard, category-specific diagnostic performance and four-week reproducibility, using intraoral digital scan exports. Methods In this retrospective diagnostic accuracy study, 90 patients were sele…

The problem
In a retrospective study of 90 patients' intraoral scans, Claude Opus 4.7, GPT-5.5 and Gemini 3 Flash agreed poorly with orthodontists (kappa up to 0.299) and detected only 10.0–13.3% of Class III cases versus 60.0–80.0% of Class I.
In brief
The chatbots were weakest on Class III malocclusion, where sensitivity was 10.0–13.3% against 60.0–80.0% for Class I. The authors conclude that overall accuracy hid this category-level weakness and that the systems are not suitable for standalone orthodontic diagnosis under the conditions tested.
Results were also unstable over time. When 20 patients were reassessed after four weeks, sagittal accuracy fell from 55.0–60.0% to 5.0–35.0%, and most test-retest kappa confidence intervals included zero.
Main points
- The study used 90 archived patient records, with 30 patients in each Angle class.
- Five standardized two-dimensional views from each iTero scan were given to each chatbot with a single zero-shot prompt.
- Two orthodontists classified each case from the complete scan record with metric measurements; their agreed categories formed the reference standard.
- Macro-average recall ranged from 38.7% (Gemini 3 Flash, arch form) to 60.9% (GPT-5.5, vertical relationship).
- Class III sensitivity was 10.0–13.3%, compared with 60.0–80.0% for Class I.
- Test-retest kappa ranged from −0.238 (Gemini 3 Flash, sagittal classification) to 0.433 (GPT-5.5, vertical relationship).
- Classifying the same five images, the orthodontists agreed with the reference standard in 96.6–100% of cases.
The full story
Researchers assessed four parameters: sagittal classification, vertical relationship, crowding severity and arch form. Accuracy, class-specific sensitivity and specificity, macro-average recall and Cohen's kappa were calculated with 95% confidence intervals.
To check whether the images themselves were the problem, the orthodontists also classified all cases from the same five images. They agreed with the reference standard in 96.6–100% of cases, and each reproduced at least 19 of 20 classifications for each parameter at retest.
The chatbots did not hold steady over time. In the 20-patient retest after four weeks, sagittal accuracy fell from 55.0–60.0% to 5.0–35.0%, and most test-retest kappa intervals included zero. The authors say reproducibility must be evaluated separately from accuracy.
Sources
1 on file- BMC Oral Healthdoi:10.1186/s12903-026-10465-5
Talk it over
The chatbots detected only 10.0% to 13.3% of Class III cases but 60.0% to 80.0% of Class I. When judging a tool like this, how would you weigh an overall accuracy figure against the result for each class?
AI chatbots misread dental scans: a 90-patient study found Class III bite problems missedhttps://truvace.com/item/large-language-models-for-orthodontic-classification-using-intraoral-digital-mod
The chatbots detected only 10.0% to 13.3% of Class III cases but 60.0% to 80.0% of Class I. When judging a tool like this, how would you weigh an overall accuracy figure against the result for each class?
Answer in the threadResearchers gave each chatbot five two-dimensional views from the iTero scans with one zero-shot prompt, while the orthodontists used the complete scan record. What can that setup tell us about how the chatbots would do with real patients?
Answer in the threadWhen 20 patients were reassessed after four weeks, sagittal accuracy fell from 55.0% to 60.0% down to 5.0% to 35.0%. What result on repeat testing would make you trust a chatbot's orthodontic classification enough to use it?
Answer in the thread
If you have more time
Which other prompts, image sets or newer chatbot versions would you want tested to decide whether the Class III weakness is built in or a product of this setup?
Answer in the threadIn a dental clinic you know, where might a chatbot classification be used as a first look, and who would check it against the orthodontists' own reading?
Answer in the thread
ace
What readers say