Assessing the accuracy of ChatGPT-5.6 in American College of Radiology Thyroid Imaging Reporting and Data System feature scoring and classification from synthetic free-text thyroid ultrasound reports

ObjectivesThe American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) provides a standardized framework for thyroid nodule risk stratification based on predefined ultrasound features. This study aimed to evaluate the accuracy and robustness of ChatGPT-5.6 Sol in extracting and scoring ACR TI-RADS features and classifying thyroid nodules from synthetic free-text ultrasound reports.MethodsA primary dataset of 100 synthetic thyroid ultrasound reports was constructed, with each report d…

By the Truvace deskOctober 16, 20262 min read1 sourcePeer-reviewedTRV-2026-1605· cite & verify

Assessing the accuracy of ChatGPT-5.6 in American College of Radiology Thyroid Imaging Reporting and Data System feature scoring and classification from synthetic free-text thyroid ultrasound reports — Ultrasound Scan ND 0107091941 0928470 by Nevit Dilmen ( talk ). CC BY-SA 3.0 · https://creativecommons.org/licenses/by-sa/3.0

In brief

Researchers tested ChatGPT-5.6 Sol on 100 synthetic thyroid ultrasound reports, each describing one nodule and the five ACR TI-RADS feature categories. The model had to identify features, score them, total the points and assign a final TI-RADS category. A separate 30-case challenge set balanced across TR1 to TR5 was also used, and outputs were compared with reference annotations based on the 2017 criteria.

The result suggests language models could help standardize how free-text ultrasound reports are converted into structured risk categories. However, the reports were synthetic rather than drawn from real clinical practice, so the findings show only that the model performed well in a controlled setting. The authors say larger and more diverse validation is needed before the results can be generalized.

Main points

The primary dataset contained 100 synthetic reports, each describing one nodule and the five ACR TI-RADS feature categories.
Feature identification and scoring were correct in 100/100 cases for every category, with a 95% confidence interval of 96.3%-100.0%.
A 30-case challenge set balanced across TR1 to TR5 also produced 30/30 correct results, with a Cohen's kappa of 1.000.
A less-directive prompt gave identical total score and classification accuracy in both datasets, with no discordant cases.

The gain

On synthetic free-text thyroid ultrasound reports, the model identified and scored all five TI-RADS feature categories and assigned the final risk category with no errors against reference annotations.

The rundown

The study used a structured prompting strategy to guide the model through feature identification, feature scoring, total score calculation and final classification. Outputs were checked against reference annotations built from the 2017 ACR TI-RADS criteria. Accuracy was reported with Wilson 95% confidence intervals, and agreement on the final category was measured with Cohen's kappa.

Robustness was tested in two ways. A less-directive prompt was used for total score calculation and final classification, and a separate challenge set spanning TR1 to TR5 was constructed. In both checks the model produced no discordant cases, though the underlying data were still synthetic.

The authors conclude that large language models may have potential for standardized TI-RADS interpretation of free-text ultrasound reports. They stop short of claiming clinical readiness and call for further validation in larger and more diverse settings.