65/100

Moderate evidence

Cite
More
EmailLinkedInX

A study of 14,848 ICU patients found a retuned risk score predicts death better

Study title: Interpretable Recalibration of a Sequential Organ Failure Assessment-Derived Score Using Machine Learning-Guided Optimization: Multicenter Cohort Study

Evidence type
Observational study
Sample
n = 14,848
Sources
1 sourcePeer-reviewed
Published
2 min read

Study title: Interpretable Recalibration of a Sequential Organ Failure Assessment-Derived Score Using Machine Learning-Guided Optimization: Multicenter Cohort Study

Researchers re-estimated the thresholds and integer weights of a SOFA-derived intensive care score from outcome data, using machine learning-guided optimization. They analysed 14,848 adult first ICU admissions from 208 US hospitals in the eICU database (2014-2015). The recalibrated score reached an AUROC of 0.811 for ICU mortality, against 0.751 for the reference score.

Read the study abstract

Background The Sequential Organ Failure Assessment (SOFA) score is central to risk stratification in intensive care units (ICUs), but its thresholds and 0- to 4-point increments were set by expert consensus in 1996 and never reestimated from outcome data. Machine learning models discriminate better while discarding the additive, point-based structure that makes SOFA auditable at the bedside, and methods for updating a legacy score rather than replacing it remain scarce. Objective This study aimed to develop and…

The gain

In a retrospective study of 14,848 first ICU admissions across 208 US hospitals, a machine learning-recalibrated SOFA-derived score predicted ICU death with an AUROC of 0.811 versus 0.751 for the reference score.

In brief

The SOFA score's cutpoints and 0- to 4-point increments were set by expert consensus in 1996 and never reestimated from outcome data. The new version keeps an additive, bedside-computable point structure on a 0-71 scale, and its discrimination came close to unconstrained XGBoost (AUROC 0.818) and exceeded penalized logistic regression (0.775).

The work is retrospective and internally validated only. Most encounters were excluded for missing data, and the authors say external and prospective validation is needed before the score is used clinically.

Main points

  1. ICU mortality in the cohort was 17.03% (2529 of 14,848 admissions).
  2. The area under the precision-recall curve rose to 0.495 from 0.393, a paired difference of 0.102.
  3. Performance came from 10-fold nested cross-validation grouped by hospital, and the improvement held across all 10 folds and length-of-stay strata.
  4. The advantage persisted in all 1000 resampling replications and in an expanded cohort of 61,469 encounters (median paired AUROC difference 0.037).
  5. The Brier score was lower (0.112 vs 0.123), but other calibration-error measures were higher.

The full story

The study used the eICU Collaborative Research Database. Only 14,848 of 148,532 encounters with outcome data had complete day 1 SOFA-related measurements; the rest were excluded for at least one missing determination. Because vasoactive dose units could not be harmonized and urine output was largely unavailable, a prespecified SOFA-derived reference score served as the comparator rather than the full SOFA.

The method pairs supervised discretization, using univariate decision trees, with constrained integer-weight optimization under monotonicity, integrality and scale-anchoring constraints, plus logistic calibration. Final thresholds and weights were refit on the full cohort and contributed no performance estimate.

The authors describe the result as a mortality risk model on its own 0-71 scale and provide an open-source calculator. They suggest the approach should transfer to other legacy clinical instruments.

Sources

1 on file
  1. Peer-reviewed2026-11-13

Talk it over

Three questions to start on, two more if you have time
The recalibrated score reached an AUROC of 0.811 against 0.751 for the reference score. How would you weigh that gain against the fact that it was only validated internally?

Start with a question

A study of 14,848 ICU patients found a retuned risk score predicts death betterhttps://truvace.com/item/interpretable-recalibration-of-a-sequential-organ-failure-assessment-derived-sco

  1. The recalibrated score reached an AUROC of 0.811 against 0.751 for the reference score. How would you weigh that gain against the fact that it was only validated internally?

    Answer in the thread
  2. Only 14,848 of 148,532 encounters with outcome data had complete day 1 measurements. Who might be missing from the patients who were left in, and how could that change the result?

    Answer in the thread
  3. What finding from an external or prospective study of 208 hospitals or others would convince you the retuned score is ready for bedside use?

    Answer in the thread
If you have more time
  • The original SOFA cutpoints were set by expert consensus in 1996. What would you want to ask about other legacy clinical instruments before retuning them the same way?

    Answer in the thread
  • The Brier score was lower (0.112 vs 0.123) but other calibration-error measures were higher. What would you ask the researchers about how that should affect use in an ICU?

    Answer in the thread

What readers say