Truace
Subscribe
68/100

Moderate evidence

Cite
More
Export CSVExport JSONEmailLinkedInX

Shandong test: AI weather model cut temperature error 13.2%, but lagged at longer lead times

Study title: Regional Evaluation of Pangu-Weather Driven by Multiple NWP Initial Fields over Shandong Province, China

Evidence type
Controlled or validated comparison
Sample
n = 123
Sources
1 sourcePeer-reviewed
Published
2 min read
Truvace record
TRV-2026-2492cite or check it

Study title: Regional Evaluation of Pangu-Weather Driven by Multiple NWP Initial Fields over Shandong Province, China

A peer-reviewed evaluation tested the AI model Pangu-Weather over Shandong Province, China, from August to December 2024. Started from ECMWF-IFS fields, it cut mean 2 m temperature error by about 13.2% compared with ECMWF operational forecasts, improving at 38 of 40 lead times, with a maximum gain of 28.3%.

Read the study abstract

Accurate weather forecasting is essential for decision-making in agriculture, transportation, disaster prevention, and energy management. In recent years, data-driven artificial intelligence (AI) models have shown considerable potential for medium-range weather forecasting by substantially improving computational efficiency while maintaining competitive forecast skill. This study evaluates the regional performance of Pangu-Weather initialized with three global numerical weather prediction (NWP) systems—ECMWF-IFS…

Regional Evaluation of Pangu-Weather Driven by Multiple NWP Initial Fields over Shandong Province, China
"Smoke from wildfire on Angel Island" by Brocken Inaglory, CC BY-SA 3.0.

57Gain side

How the two sides scoreClose call

61Problem side

The gain

In a five-month test against 123 weather stations in Shandong, Pangu-Weather started from ECMWF data cut 2 m temperature error by about 13.2% against ECMWF's own operational forecasts, improving at 38 of 40 lead times.

The problem

In the same Shandong test, Pangu-Weather runs started from CMA and NCEP data had larger errors than ECMWF forecasts at several longer lead times over coastal stations and at 500–700 hPa.

A fixed formula sets both scores from the evidence and its sources. Reader votes don’t move them.

What the scores mean

G is the gain side, out of 100. It goes up when the gain is specific, when it measurably helped people and when the sources behind it are strong.

P is the problem side, out of 100. It goes up when the problem is specific, when it measurably hurt people and when the sources behind it are strong.

In brief

Surface forecasts were checked against 123 national meteorological stations and upper-air forecasts against three radiosonde stations. Wind-speed error at 10 m fell at every lead time, by 5.7% on average and up to 10.6%. Specific-humidity forecasts improved in all three configurations, by 8.6–12.8% on average, with a peak of 33.7% in one comparison.

Results were not uniform. Runs started from CMA-GFS and NCEP-GFS fields had larger errors than ECMWF at several longer lead times over coastal stations and at 500–700 hPa, and upper-air temperature and wind gains varied by level, station and lead time. The authors say the model could serve as a regional forecasting aid, with performance depending on the initial fields, variable, location and lead time.

Main points

  1. The test covered August to December 2024 over Shandong Province, China.
  2. Pangu-Weather was started from three global systems: ECMWF-IFS, CMA-GFS and NCEP-GFS.
  3. Surface forecasts were verified against 123 national meteorological stations; upper-air forecasts against three radiosonde stations.
  4. The ECMWF-initialized run cut 2 m temperature error by about 13.2%, improving at 38 of 40 lead times.
  5. Its 10 m wind-speed error was lower at every lead time, by 5.7% on average and 10.6% at most.
  6. Specific-humidity forecasts improved in all three configurations, by 8.6–12.8% on average.
  7. CMA- and NCEP-initialized runs were worse than ECMWF at several longer lead times over coastal stations and at 500–700 hPa.

The full story

The study assessed how Pangu-Weather, a data-driven AI model for medium-range forecasting, performs at regional scale when fed different operational starting fields. It compared runs from ECMWF-IFS, CMA-GFS and NCEP-GFS with ECMWF operational forecasts over Shandong Province.

The ECMWF-initialized run (EC_PG) gave the most robust overall performance. Its mean 2 m temperature error was about 13.2% lower than ECMWF operational forecasts, with a maximum improvement of 28.3%. Its 10 m wind-speed error was lower at every lead time, with a mean improvement of 5.7% and a maximum of 10.6%.

Upper-air results were mixed. Specific-humidity forecasts generally improved across three stations and three pressure levels, by 8.6–12.8% on average and up to 33.7% in one comparison. Temperature and wind improvements depended on pressure level, station and lead time, and the CMA- and NCEP-started runs trailed ECMWF at some longer lead times.

What this doesn’t fix

Performance depended on the initial fields, variable, location and lead time, and upper-air temperature and wind gains varied by pressure level, station and lead time.

Sources

1 on file
  1. Atmospheredoi:10.3390/atmos17111087
    Peer-reviewed2026-11-04

Talk it over

Four questions to start onThis record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

Start with a question

Shandong test: AI weather model cut temperature error 13.2%, but lagged at longer lead timeshttps://truvace.com/item/regional-evaluation-of-pangu-weather-driven-by-multiple-nwp-initial-fields-over-

  1. This record found a gain and a problem in the same place. Which matters more to you, and who might answer differently?

    The gain In a five-month test against 123 weather stations in Shandong, Pangu-Weather started from ECMWF data cut 2 m temperature error by about 13.2% against ECMWF's own operational forecasts, improving at 38 of 40 lead times.

    The problem In the same Shandong test, Pangu-Weather runs started from CMA and NCEP data had larger errors than ECMWF forecasts at several longer lead times over coastal stations and at 500–700 hPa.

    Answer in the thread

  2. How do we know? What can this kind of evidence show, and what can’t it?

    The evidence This rests on a controlled comparison with a sample of 123, from Atmosphere.

    Answer in the thread

  3. What does it leave open? What else would you want to know before acting on it?

    The record’s own caveat Performance depended on the initial fields, variable, location and lead time, and upper-air temperature and wind gains varied by pressure level, station and lead time.

    Answer in the thread

  4. Have you seen AI used this way where you study, work or live? Did it go the same way?

    Answer in the thread

The questions are the same on every record. The details under them are taken from this record as written.

What readers say

More from Climate