Moderate evidence
More
Only 17.89% of chest X-ray AI flags were right, a study of 826 Nepal applicants found
Study title: Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal
- Evidence type
- Controlled or validated comparison
- Sample
- n = 826
- Sources
- 1 sourcePeer-reviewed
- Published
- 1 min read
- Truvace record
- TRV-2026-1265cite or check it
Study title: Diagnostic accuracy of a DenseNet-121 deep learning algorithm for chest radiograph triage in health assessment applicants: a prospective shadow-mode validation study in Nepal
From 5 to 20 June 2026, researchers prospectively tested a publicly available DenseNet-121 model from TorchXRayVision in shadow mode on 826 consecutive chest radiographs from health assessment applicants at Patan Hospital in Lalitpur, Nepal, comparing AI scores to blinded single-reader radiologist classifications.
Read the study abstract
Objectives To evaluate the diagnostic accuracy of a publicly available DenseNet-121 convolutional neural network (TorchXRayVision) for triaging chest radiographs of health assessment applicants at a tertiary hospital in Nepal. Design Prospective, single-centre, shadow-mode diagnostic accuracy validation study. Reported in accordance with the Standards for Reporting of Diagnostic Accuracy Studies (STARD) 2015 checklist and STARD-Artificial Intelligence (AI)/Developmental and Exploratory Clinical Investigations of…
The problem
At a threshold set for at least 95% sensitivity, a study of 826 chest X-rays in Nepal found only 17.89% of the AI's abnormal flags were correct, agreement with radiologists was low (kappa 0.237), and its probability scores were badly miscalibrated (ECE 0.564).
In brief
The model preserved discrimination with high sensitivity and NPV, suggesting rule-out triage potential, but exhibited systematic calibration failure and low PPV in this population, so threshold stability, the strength of the reference standard and readiness for operational deployment remain uncertain without local recalibration and independent external validation.
Main points
- Prospective single-centre shadow-mode study from 5 June 2026 to 20 June 2026 inclusive with 826 consecutive applicants; 41 (4.97%) classified abnormal by reference standard.
- Index test was TorchXRayVision densenet121-res224-all with maximum aggregated pathology probability compared against post hoc threshold 0.6258 selected for >=95% sensitivity.
- Reference standard was single-reader-per-case review by two board-certified radiodiagnosticians and one resident, blinded to AI output.
- 10-fold cross-validation showed bias-corrected specificity 75.80% with optimism +1.40 pp; independent external validation was not performed.
The full story
The study enrolled 826 consecutive foreign employment predeparture and student migration applicants at Patan Academy of Health Sciences/Patan Hospital, with two exclusions for DICOM technical failure, and used a maximum aggregated pathology probability score.
At threshold 0.6258, specificity was 77.2% (95% CI 74.1% to 80.0%) with PPV 17.89% and Cohen's kappa 0.237, while Brier score 0.3621 exceeded null Brier 0.0472 and ECE was 0.564, showing score compression from 0.52-0.72.
Authors concluded high sensitivity supports potential as radiographic abnormality rule-out triage tool but noted this does not constitute microbiological exclusion of active pulmonary tuberculosis and called for prospective local calibration and external validation.
Sources
1 on file- BMJ Opendoi:10.1136/bmjopen-2026-124868
Talk it over
At a threshold set for at least 95% sensitivity, only 17.89% of the AI's abnormal flags were correct among 826 applicants. How would you weigh catching nearly every abnormal film against so many wrong flags?
Only 17.89% of chest X-ray AI flags were right, a study of 826 Nepal applicants foundhttps://truvace.com/item/diagnostic-accuracy-of-a-densenet-121-deep-learning-algorithm-for-chest-radiogra
At a threshold set for at least 95% sensitivity, only 17.89% of the AI's abnormal flags were correct among 826 applicants. How would you weigh catching nearly every abnormal film against so many wrong flags?
Answer in the threadOnly 41 of the 826 X-rays were classed abnormal, and one radiologist reviewed each case. How much would you trust the 17.89% figure with so few positives, and what would make it firmer?
Answer in the threadImagine an independent external validation at another hospital. What result there would convince you the model is ready for operational use at Patan Hospital, and what would settle it against?
Answer in the thread
If you have more time
The authors called for prospective local calibration. Who at a hospital would need to carry that out, and what would you ask them to show before the tool is used?
Answer in the threadThe study notes the AI result is not a microbiological exclusion of active pulmonary tuberculosis. How might applicants flagged wrongly or missed be affected differently in practice?
Answer in the thread
ace
What readers say