55/100

Moderate evidence

Cite
More
EmailLinkedInX

Dental X-ray test finds AI chatbots miss low-contrast findings and err on decisions

Study title: Cognitive Evaluation of Multimodal Large Language Models in Dental Radiography

Evidence type
Benchmark or lab evaluation
Sample
n = 3,820
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Cognitive Evaluation of Multimodal Large Language Models in Dental Radiography

Researchers tested six general-purpose multimodal AI models on dental X-rays using a new benchmark, OralMLLM-Bench, with 27 tasks across periapical, panoramic and lateral cephalometric images. Two clinician comparators were also assessed, across 3,820 assessment units in all.

Read the study abstract

Objectives: We evaluated the cognitive capabilities and safety boundaries of general-purpose, zero-shot multimodal large language models (MLLMs) in dental radiography. Methods: OralMLLM-Bench comprised 27 independently evaluated tasks across periapical, panoramic, and lateral cephalometric radiographs, grouped into four categories: perception, comprehension, prediction, and decision-making. Six MLLMs and two clinician comparators were evaluated across 3,820 assessment units using sensitivity, specificity, balanc…

The problem

In a benchmark of six multimodal AI models on 3,820 dental radiograph assessment units, performance was weaker for low-contrast findings and spatial localization, and agreement with expert references was generally limited for prediction and decision-making tasks.

In brief

The models did best on visually obvious findings, with sensitivity up to 0.917 for root filling detection and 0.875 for restoration detection. They did worse on low-contrast findings and on locating structures in space.

Only 6 of the 27 tasks showed significant differences between models after false discovery rate correction, and no model had a broad, consistent advantage. Errors included false positives caused by image artifacts, false negatives on low-contrast findings, and conclusions that did not match the model's own rationale, as well as uncertainty the models failed to flag.

Main points

  1. The benchmark, OralMLLM-Bench, has 27 independently evaluated tasks on periapical, panoramic and lateral cephalometric radiographs.
  2. Tasks fall into four categories: perception, comprehension, prediction and decision-making.
  3. Six multimodal AI models and two clinician comparators were assessed across 3,820 assessment units.
  4. Sensitivity reached 0.917 for root filling detection and 0.875 for restoration detection.
  5. Only 6 of 27 tasks showed significant between-model differences after false discovery rate correction.
  6. Agreement with expert image-based reference assessments was generally limited in prediction and decision-making tasks.
  7. Observed errors included artifact-induced false positives, low-contrast false negatives, spatial localization errors, conclusion-rationale inconsistency and unflagged diagnostic uncertainty.

The full story

The study evaluated general-purpose, zero-shot multimodal models, meaning none were specially trained for dental imaging. They were scored on sensitivity, specificity, balanced accuracy, a partial-credit average score and conditional rationale accuracy, which checks whether a model's explanation holds up.

Results varied by task. Models handled salient findings such as root fillings and restorations comparatively well, but struggled where contrast was low or where the answer required pinpointing location, and they fared worst on prediction and decision-making tasks.

The authors conclude that under standardized, image-only conditions the models showed task-dependent performance and persistent limitations, and offer the benchmark as a framework for judging the safety boundaries of such systems.

Sources

1 on file
  1. Journal of Dentistrydoi:10.1016/j.jdent.2026.107174
    Peer-reviewed2026-11-15

Talk it over

Three questions to start on, two more if you have time
The six general-purpose models did best on root fillings (sensitivity up to 0.917) and worse on low-contrast findings. How would you weigh the easy wins against the misses when judging whether such a tool is safe to use?

Start with a question

Dental X-ray test finds AI chatbots miss low-contrast findings and err on decisionshttps://truvace.com/item/cognitive-evaluation-of-multimodal-large-language-models-in-dental-radiography

  1. The six general-purpose models did best on root fillings (sensitivity up to 0.917) and worse on low-contrast findings. How would you weigh the easy wins against the misses when judging whether such a tool is safe to use?

    Answer in the thread
  2. The benchmark used standardized, image-only conditions across 27 tasks. How might a dentist's usual access to a patient's history and exam change what these results tell us?

    Answer in the thread
  3. Only 6 of the 27 tasks showed significant differences between models after false discovery rate correction. What kind of finding would convince you that one model deserves trust on prediction or decision-making tasks?

    Answer in the thread
If you have more time
  • The researchers saw conclusions that did not match a model's own rationale and uncertainty the models failed to flag. How would you look up or test whether a dental tool shows its doubt to the person using it?

    Answer in the thread
  • The models were not specially trained for dental imaging. What would you want to compare next to see whether dental-specific training closes the gaps on low-contrast findings and spatial localization?

    Answer in the thread

What readers say