Moderate evidence
More
A study of 24,434 chest CT reports finds AI-cleaned labels beat the originals
Study title: Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset
- Evidence type
- Controlled or validated comparison
- Sample
- n = 24,434
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-3088cite or check it
Study title: Large Language Model-Assisted Cleaning of Report-Derived Labels in a Large-Scale Chest CT Dataset
Researchers used GPT-5.4 to re-derive yes/no labels for 18 abnormality categories from 24,434 radiology reports in CT-RATE, a large public chest CT dataset, and compared them with the original labels. Overall agreement was 96.4%, with a Cohen's κ of 0.884, and radiologists adjudicated the disagreements.
Read the study abstract
To evaluate whether large language model (LLM)-assisted label cleaning can identify label-report discordance in CT-RATE, a large-scale public chest CT dataset. After report-level deduplication, 24,446 unique radiology reports from 20,209 unique patients were identified. Twelve reports were excluded from the primary GPT-5.4 analysis because of persistent content-safety filtering or unparsable output, leaving 24,434 reports and 439,812 label instances across 18 abnormality categories. GPT-5.4-derived binary labels…

The gain
In a study of 24,434 chest CT reports, a three-model AI majority vote scored an F1 of 95.6% against two blinded radiologists on 250 reports, versus 90.8% for the dataset's original labels.
In brief
In a prespecified confirmatory set of 250 reports annotated independently by two blinded radiologists, a three-model majority vote reached an F1 of 95.6% versus 90.8% for the original labels, a 4.8-point gain (95% CI, 3.6-6.2). Each single model also beat the original labels.
Lymphadenopathy showed the lowest agreement, so that category had the most label-report discordance. The labels reflect what the reports say, not what the images show, so they do not confirm image-level truth.
Main points
- The analysis covered 24,434 reports from 20,209 patients and 439,812 label instances across 18 abnormality categories; 12 reports were excluded for content-safety filtering or unparsable output.
- GPT-5.4 labels agreed with the original CT-RATE labels 96.4% of the time (κ = 0.884), with lymphadenopathy showing the lowest agreement and κ.
- On the 250-report confirmatory set, the majority vote scored an F1 of 95.6% and κ of 0.940, versus 90.8% and 0.879 for the original labels.
- Pre-consensus agreement between the two blinded radiologists was κ = 0.936, and 1.02% of label instances flipped across three repeated majority-vote runs.
- The refined labels and code are to be made public upon acceptance.
The full story
The team deduplicated the dataset to 24,446 unique reports, then generated binary labels for each of 18 abnormality categories using GPT-5.4 with structured outputs. Radiologists adjudicated the instances where the AI labels and the original labels disagreed.
A 100-report random development set was manually annotated to compare the original labels, individual model labels and three-model majority-vote labels against radiologist reference labels. The methods were then tested on a stratified confirmatory set of 250 reports annotated independently by two blinded radiologists.
The majority vote improved on the original labels by 4.8 F1 points, and the run-to-run flip rate of 1.02% suggests the output was largely stable. The authors say the approach may offer a scalable way to improve the quality of public imaging datasets.
Sources
1 on file- Journal of Medical Systemsdoi:10.1007/s10916-026-02487-x
Talk it over
The labels in CT-RATE came from what the reports say, and the 250 reports were checked by two blinded radiologists. How much should that tell us about whether the AI-cleaned labels match what the scans show?
A study of 24,434 chest CT reports finds AI-cleaned labels beat the originalshttps://truvace.com/item/large-language-model-assisted-cleaning-of-report-derived-labels-in-a-large-scale
The labels in CT-RATE came from what the reports say, and the 250 reports were checked by two blinded radiologists. How much should that tell us about whether the AI-cleaned labels match what the scans show?
Answer in the threadThe three-model majority vote reached an F1 of 95.6% versus 90.8% for the original labels. Which would you weigh more: that 4.8-point gain, or the 12 reports excluded and the 1.02% of label instances that flipped between runs?
Answer in the threadIf you were one of the radiologists adjudicating the disagreements between GPT-5.4 and the original labels, how would you decide whose label to trust, especially for lymphadenopathy?
Answer in the thread
If you have more time
Look up how researchers who build public imaging datasets check label quality now. What would you compare against the 250-report confirmatory set?
Answer in the threadIf the refined labels are made public, who should check them, and what finding in a new set of reports would settle whether the cleaning holds up?
Answer in the thread
ace
What readers say