68/100

Moderate evidence

Cite
More
EmailLinkedInX

Only 17.89% of chest X-ray AI flags were right, a study of 826 Nepal applicants found

Study title: Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal

Evidence type
Controlled or validated comparison
Sample
n = 826
Sources
1 sourcePeer-reviewed
Published
1 min read

Study title: Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal

From 5 to 20 June 2026, researchers prospectively tested a publicly available DenseNet-121 model from TorchXRayVision in shadow mode on 826 consecutive chest radiographs from health assessment applicants at Patan Hospital in Lalitpur, Nepal, comparing AI scores to blinded single-reader radiologist classifications.

Read the study abstract

Objectives To evaluate the diagnostic accuracy of a publicly available DenseNet-121 convolutional neural network (TorchXRayVision) for triaging chest radiographs of health assessment applicants at a tertiary hospital in Nepal. Design Prospective, single-centre, shadow-mode diagnostic accuracy validation study. Reported in accordance with the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 checklist and STARD-Artificial Intelligence (AI)/Developmental and Exploratory Clinical Investigations of…

Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal
Photo by Fotos on Unsplash · https://unsplash.com/@fotospk?utm_source=truvace&utm_medium=referral

The problem

At a threshold set for at least 95% sensitivity, a study of 826 chest X-rays in Nepal found only 17.89% of the AI's abnormal flags were correct, agreement with radiologists was low (kappa 0.237), and its probability scores were badly miscalibrated (ECE 0.564).

In brief

The model preserved discrimination with high sensitivity and NPV, suggesting rule-out triage potential, but exhibited systematic calibration failure and low PPV in this population, so threshold stability, the strength of the reference standard and readiness for operational deployment remain uncertain without local recalibration and independent external validation.

Main points

  1. Prospective single-centre shadow-mode study from 5 June 2026 to 20 June 2026 inclusive with 826 consecutive applicants; 41 (4.97%) classified abnormal by reference standard.
  2. Index test was TorchXRayVision densenet121-res224-all with maximum aggregated pathology probability compared against post hoc threshold 0.6258 selected for >=95% sensitivity.
  3. Reference standard was single-reader-per-case review by two board-certified radiodiagnosticians and one resident, blinded to AI output.
  4. 10-fold cross-validation showed bias-corrected specificity 75.80% with optimism +1.40 pp; independent external validation was not performed.

The full story

The study enrolled 826 consecutive foreign employment predeparture and student migration applicants at Patan Academy of Health Sciences/Patan Hospital, with two exclusions for DICOM technical failure, and used a maximum aggregated pathology probability score.

At threshold 0.6258, specificity was 77.2% (95% CI 74.1% to 80.0%) with PPV 17.89% and Cohen's kappa 0.237, while Brier score 0.3621 exceeded null Brier 0.0472 and ECE was 0.564, showing score compression from 0.52-0.72.

Authors concluded high sensitivity supports potential as radiographic abnormality rule-out triage tool but noted this does not constitute microbiological exclusion of active pulmonary tuberculosis and called for prospective local calibration and external validation.

Sources

1 on file
  1. BMJ Opendoi:10.1136/bmjopen-2026-124868
    Peer-reviewed2026-10-01

Talk it over

Three questions to start on, two more if you have time
At a threshold set for at least 95% sensitivity, only 17.89% of the AI's abnormal flags were correct among 826 applicants. How would you weigh catching nearly every abnormal film against so many wrong flags?

Start with a question

Only 17.89% of chest X-ray AI flags were right, a study of 826 Nepal applicants foundhttps://truvace.com/item/diagnostic-accuracy-of-a-densenet-121-deep-learning-algorithm-for-chest-radiogra

  1. At a threshold set for at least 95% sensitivity, only 17.89% of the AI's abnormal flags were correct among 826 applicants. How would you weigh catching nearly every abnormal film against so many wrong flags?

    Answer in the thread
  2. Only 41 of the 826 X-rays were classed abnormal, and one radiologist reviewed each case. How much would you trust the 17.89% figure with so few positives, and what would make it firmer?

    Answer in the thread
  3. Imagine an independent external validation at another hospital. What result there would convince you the model is ready for operational use at Patan Hospital, and what would settle it against?

    Answer in the thread
If you have more time
  • The authors called for prospective local calibration. Who at a hospital would need to carry that out, and what would you ask them to show before the tool is used?

    Answer in the thread
  • The study notes the AI result is not a microbiological exclusion of active pulmonary tuberculosis. How might applicants flagged wrongly or missed be affected differently in practice?

    Answer in the thread

What readers say