aceModerate evidence
More
Hail, wind and tornado forecasts change sharply with training data, study finds
Study title: The Impact of Target Dataset Choice on Machine Learning Prediction of Severe Hail, Wind, and Tornadoes
- Evidence type
- Benchmark or lab evaluation
- Sources
- 1 sourcePeer-reviewed
- Published
- 2 min read
- Truvace record
- TRV-2026-2543cite or check it
Study title: The Impact of Target Dataset Choice on Machine Learning Prediction of Severe Hail, Wind, and Tornadoes
Researchers compared two sets of random forest models that forecast severe hail, wind and tornado hazards one hour at a time, at lead times from 0 to 3 hours. One set was trained on storm reports expanded in space and time; the other was trained on hazard-specific National Weather Service warnings.
Read the study abstract
Abstract Supervised machine learning (ML) methods implicitly assume that their target dataset is error free. For severe weather prediction, point-based observed Storm Data storm reports that have been expanded in space and time are commonly used targets. However, other target datasets could potentially better represent severe weather occurrence, especially for situations requiring more precision. This study analyzes 1-h probabilistic severe weather hazard forecasts made at lead times from 0 to 3 h from random fo…
The problem
In a comparison of random forest models, severe weather forecasts trained on National Weather Service warnings scored notably worse when checked against storm reports than report-trained models checked against warnings.
In brief
Forecast probabilities came out drastically different depending on the target. Warnings-trained models gave higher probabilities after 0000 UTC and lower ones before it than report-trained models, and their forecasts were sharper and more spatially focused, especially for hail and wind.
Which model looked better depended on the yardstick. Models trained and verified on warnings scored better, but warnings-trained models verified on reports scored notably worse than report-trained models verified on warnings. The authors say developers and users of ML forecasts need to weigh the target dataset when building models and reading results.
Main points
- All models used predictors from the Warn-on-Forecast System, ProbSevere version 3 and the tornado probability algorithm; only the training target differed.
- Both target datasets were built to have the same overall climatological frequency for each hazard but different spatial and temporal patterns.
- Models tended to reproduce the climatology of their target: warnings-trained models gave higher probabilities after 0000 UTC and lower ones before it.
- Warnings-trained forecasts were sharper and more spatially focused, especially for severe hail and wind.
- Models trained and verified on warnings scored better than report-trained models on the same verification.
The full story
The study tested a common assumption in supervised machine learning, that the target dataset is error free. For severe weather, the usual target is point-based Storm Data reports spread over surrounding space and time, which may not capture precisely where and when hazards occurred.
To test alternatives, the authors trained random forests on either those expanded reports or hazard-specific National Weather Service warnings, holding everything else constant, and produced 1-hour probabilistic forecasts at lead times of 0 to 3 hours.
The cross-verification result is the key caution: warnings-trained forecasts checked against reports did notably worse than report-trained forecasts checked against warnings. The authors conclude that target dataset selection matters during both model development and evaluation.
Sources
1 on file- Artificial Intelligence for the Earth Systemsdoi:10.1175/aies-d-26-0004.1
Talk it over
Who carries this cost, and what would have to change to prevent it?
Hail, wind and tornado forecasts change sharply with training data, study findshttps://truvace.com/item/the-impact-of-target-dataset-choice-on-machine-learning-prediction-of-severe-hai
Who carries this cost, and what would have to change to prevent it?
The problem In a comparison of random forest models, severe weather forecasts trained on National Weather Service warnings scored notably worse when checked against storm reports than report-trained models checked against warnings.
Answer in the threadHow do we know? What can this kind of evidence show, and what can’t it?
The evidence This rests on a benchmark or lab test, from Artificial Intelligence for the Earth Systems.
Answer in the threadWhat does it leave open? What else would you want to know before acting on it?
The record’s own caveat Verification results depend on which dataset is used as truth, since the two target datasets have different spatial and temporal climatologies, so neither model clearly outperforms the other overall.
Answer in the threadHave you seen AI used this way where you study, work or live? Did it go the same way?
Answer in the thread
The questions are the same on every record. The details under them are taken from this record as written.
What readers say