Moderate evidence
More
Diabetes risk model for psychiatric patients held up for six years, study finds
Study title: Temporal stability of an electronic health record-driven machine learning model predicting type 2 diabetes among patients with mental illness
- Evidence type
- Observational study
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2891cite or check it
Study title: Temporal stability of an electronic health record-driven machine learning model predicting type 2 diabetes among patients with mental illness
Researchers tested whether a machine learning model that predicts type 2 diabetes in people with mental illness stays accurate over time. The XGBoost model, trained on electronic health record data from 2013 to 2017, was applied to patients with psychiatric service contacts in each year from 2018 to 2023.
Read the study abstract
We previously trained and tested a machine learning model to predict incident type 2 diabetes (T2D) in patients with mental illness using electronic health record (EHR) data from routine clinical practice. This hyperparameter-tuned XGBoost model - based on information on sex, age and mean HbA1c within the past two years - predicts whether a patient will meet criteria for T2D in the two years following each contact to psychiatric services. The performance of such a model may drift over time due to changes in stan…

The gain
In a temporal validation study, an XGBoost model trained on 2013-2017 health records predicted type 2 diabetes two years ahead in psychiatric patients with AUROC of 0.80 to 0.87 across contacts in 2018 to 2023, declining only 0.004 per year.
In brief
The model's AUROC was 0.84 in 2018, 0.83 in 2019, 0.87 in 2020, 0.80 in 2021, 0.84 in 2022 and 0.80 in 2023. A linear model estimated a yearly drop of only 0.004, with a 95% confidence interval of -0.024 to 0.016, so a meaningful decline cannot be ruled out.
The authors say the stability supports clinical implementation for early detection and tailored interventions, while stressing continued evaluation. Secondary analyses suggested that models trained on the past year of data may discriminate as well as or better than those trained on longer histories.
Main points
- The model uses only three inputs: sex, age and mean HbA1c over the past two years.
- It predicts whether a patient will meet type 2 diabetes criteria in the two years after each psychiatric service contact.
- It was fitted on records from January 1, 2013 to December 31, 2017 and tested on contacts in 2018 through 2023.
- Yearly AUROC ranged from 0.80 (2021 and 2023) to 0.87 (2020).
- The estimated AUROC decline was 0.004 per year (95% CI: -0.024 to 0.016).
- Models trained on data from the past year may match or beat models trained on longer historical periods.
The full story
The model is a hyperparameter-tuned XGBoost classifier previously built on routine clinical data. The new study asks whether its performance drifts because of changes in standards of care, baseline risk or registration procedures.
Performance stayed in the good range every year, from 0.80 to 0.87, with no clear downward trend. The wide confidence interval around the yearly change means the data are compatible with both a small improvement and a small decline.
The authors conclude that the findings support potential clinical implementation and recommend continued evaluation, and that retraining on recent data could be an option.
Sources
1 on file- Acta Neuropsychiatricadoi:10.1017/neu.2026.10105
Talk it over
The model's AUROC ranged from 0.80 to 0.87 across 2018 to 2023, but the 95% confidence interval for the yearly change ran from -0.024 to 0.016. How would you weigh the steady scores against that uncertainty?
Diabetes risk model for psychiatric patients held up for six years, study findshttps://truvace.com/item/temporal-stability-of-an-electronic-health-record-driven-machine-learning-model-
The model's AUROC ranged from 0.80 to 0.87 across 2018 to 2023, but the 95% confidence interval for the yearly change ran from -0.024 to 0.016. How would you weigh the steady scores against that uncertainty?
Answer in the threadThe model uses only sex, age and mean HbA1c over the past two years. Which patients with mental illness might this simple set of inputs serve less well, and why?
Answer in the threadIf you were a clinician deciding whether to use this model after a psychiatric contact, what would you want to see in the yearly results before relying on it, and who should check it?
Answer in the thread
If you have more time
Look up how a hospital you know checks whether a prediction tool still works after standards of care or registration procedures change. How does that compare with the yearly testing here?
Answer in the threadSecondary analyses suggested models trained on the past year of data may match or beat those trained on longer histories. What result would settle whether retraining on recent data is worth doing?
Answer in the thread
ace
What readers say