Truace
Subscribe
68/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

A US test found machine learning spots ice pellet zones the GEFS forecast misses

Study title: Relative Robustness of Machine Learning (ML) Postprocessing Schemes in Forecasting Wintertime Precipitation Types: A CONUS-Wide Comparison

Evidence type
Controlled or validated comparison
Sources
1 sourcePeer-reviewed
Published
3 min read
Truvace record
TRV-2026-2702cite or check it

Study title: Relative Robustness of Machine Learning (ML) Postprocessing Schemes in Forecasting Wintertime Precipitation Types: A CONUS-Wide Comparison

Researchers from the University of Texas at Arlington, NOAA and partner organisations compared three machine-learning methods (multinomial logistic regression, a neural network and quantile random forest) for correcting medium-range winter precipitation-type forecasts. They tested them against the operational GEFS real-time product using crowdsourced mPING reports from November–March 2017–19 across the contiguous US.

Read the study abstract

Abstract Accurate forecasts of precipitation type (p-type), rain (RN), snow (SN), ice pellets (IP), and freezing rain (FZRA) are essential for managing winter weather impacts. Operational p-type products, diagnosed from numerical weather prediction (NWP) models using physical schemes, perform reliably for common classes but fail to capture minority classes (IP/FZRA). This study evaluates three machine learning (ML) techniques, such as multinomial logistic regression (MLR), artificial neural network (ANN), and qu…

55Gain side

How the two sides scoreClose call

55Problem side

The gain

In a test on November–March 2017–19 data across the contiguous US, a multinomial logistic regression postprocessing model beat the operational GEFS forecast for ice pellets in both accuracy and equitable threat score, while GEFS showed negligible skill.

The problem

In a contiguous-US test of machine-learning postprocessing of GEFS winter forecasts, all models were poorly calibrated for ice pellets and freezing rain, giving high probabilities to events that occurred far less often.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

The operational product handled rain and snow reliably but almost never forecast ice pellets and showed freezing rain only in small, isolated patches, missing icing corridors. The machine-learning models outlined those corridors more faithfully, and logistic regression beat every other scheme for ice pellets in accuracy and equitable threat score.

The gains came with costs. For snow, the GEFS baseline kept the highest categorical accuracy. For freezing rain and ice pellets, all models were overconfident, assigning high probabilities to events that occurred much less often, and the GEFS product scored the highest equitable threat score for freezing rain by issuing fewer, safer forecasts.

Main points

  1. The models were trained on GEFSv12 reforecast data (five ensemble members, 33 predictors) paired with mPING observations, at 6-hourly lead times from 6 to 120 hours.
  2. Evaluation used three-fold cross-validation by whole year, training on two winters and testing on the third, so models were scored on years they had not seen.
  3. For rain, all three ML models beat both GEFS baselines up to 72 hours; accuracy was about 95% at 24 hours, and the logistic regression and neural network scored equitable threat above 0.9 at 24 hours.
  4. Rain skill fell with lead time: logistic regression and neural network equitable threat dropped to about 0.40 by 120 hours, quantile random forest from about 0.78 to 0.28, while the GEFS ensemble mean held near 0.47–0.48.
  5. For snow, the GEFS ensemble mean had the highest accuracy, rising from about 0.74 to nearly 0.87 at 120 hours, with quantile random forest the best ML model.
  6. For freezing rain, ML models had higher raw accuracy (the neural network peaked near 0.34 at 120 hours), yet GEFS had the highest equitable threat score because its conservative forecasts avoided false alarms.
  7. For ice pellets, logistic regression led in accuracy and equitable threat score, the neural network came second, and GEFS showed negligible skill.
  8. Case studies of three winter events, one each from 2017, 2018 and 2019, supported the finding that GEFS under-resolves mixed-phase transition zones.

The full story

The operational comparison was GEFS's real-time p-type product, which diagnoses a single dominant type from wet-bulb temperature profiles. During 2017–19 it ran at 1-degree resolution with one control and 20 perturbed members, while the ML predictors came from 0.25-degree reforecasts. The authors say the coarse vertical resolution and threshold logic smooth shallow melting and refreezing layers, which leads to underrepresentation of ice pellets and freezing rain.

Observations came from mPING, a crowdsourced app, with reports mapped to rain, snow, freezing rain and ice pellets. Reports of solid or freezing types where surface wet-bulb temperature exceeded 1°C were removed. Test folds were heavily imbalanced: in the first fold, for example, the average test set held about 8,801 ice pellet and 8,462 freezing rain samples against 42,924 rain and 42,472 snow. Inverse-frequency class weights were used in training.

On probability quality, the neural network had the lowest Brier scores among ML models for rain, snow, freezing rain and ice pellets, with logistic regression close behind and quantile random forest lagging. GEFS's low Brier scores for rare classes largely reflected near-zero probabilities regardless of outcomes. Reliability diagrams showed ANN and MLR close to the diagonal for rain at 120 hours but underconfident for snow at mid-probabilities.

The authors conclude that GEFS's modest calibration gains for freezing rain and ice pellets at long leads are illusory and do not amount to detection skill, while the ML schemes, despite overforecasting, flagged more rare-event locations.

What this doesn’t fix

The ML models overpredicted ice pellets and freezing rain, and the observations came from crowdsourced mPING reports that can be spurious and inconsistent. The baseline also ran at coarser resolution than the ML predictors.

Sources

1 on file
  1. Weather and Forecastingdoi:10.1175/waf-d-25-0205.1
    Peer-reviewed2026-06-05

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

A US test found machine learning spots ice pellet zones the GEFS forecast misseshttps://truvace.com/item/relative-robustness-of-machine-learning-ml-postprocessing-schemes-in-forecasting

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a test on November–March 2017–19 data across the contiguous US, a multinomial logistic regression postprocessing model beat the operational GEFS forecast for ice pellets in both accuracy and equitable threat score, while GEFS showed negligible skill.

    The problem In a contiguous-US test of machine-learning postprocessing of GEFS winter forecasts, all models were poorly calibrated for ice pellets and freezing rain, giving high probabilities to events that occurred far less often.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a controlled comparison, from Weather and Forecasting.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat The ML models overpredicted ice pellets and freezing rain, and the observations came from crowdsourced mPING reports that can be spurious and inconsistent.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Health