Competition Overview

Table of Contents

  1. Announcements
  2. Description
  3. Timeline
  4. Prize
  5. Predictions
  6. Evaluation
  7. Data
  8. Training
  9. Discussions
  10. Leaderboard
  11. Rules
  12. Organizers

Announcements

2021-05-10: Rules adapted to discourage overfitting

The organizers modified the rules:

The organizers are aware that overfitting is an issue if the ground truth is accessible. A more robust verification would require predicting future states with weekly submissions over a year, which would take much more time until one year of new observations is available. Therefore, we decided against a real-time competition to shorten the project length and keep momentum high. Over time we will all see which methods genuinely have skill and which overfitted their available data.

Description

The World Meteorological Organization (WMO) is launching an open prize challenge to improve current forecasts of precipitation and temperature from today’s best computational fluid dynamical models 3 to 6 weeks into the future using Artificial Intelligence and/or Machine Learning techniques. The challenge is part of the the Subseasonal-to-Seasonal Prediction Project (S2S Project), coordinated by the World Weather Research Programme (WWRP)/World Climate Research Programme (WCRP), in collaboration with Swiss Data Science Center (SDSC) and European Centre for Medium-Range Weather Forecasts (ECMWF).

Improved sub-seasonal to seasonal (S2S) forecast skill would benefit multiple user sectors immensely, including water, energy, health, agriculture and disaster risk reduction. The creation of an extensive database of S2S model forecasts has provided a new opportunity to apply the latest developments in machine learning to improve S2S prediction of temperature and total precipitation forecasts up to 6 weeks ahead, with focus on biweekly averaged conditions around the globe.

The competition will be implemented on the platform of Renkulab at the Swiss Data Science Center (SDSC), which hosts all the codes and scripts. The training and verification data will be easily accessible from the European Weather Cloud and relevant access scripts will be provided to the participants. All the codes and forecasts of the challenge will be made open access after the end of the competition.

This is the landing page of the competition presenting static information about the competition. For code examples and how to contribute, please visit the contribution template repository renkulab.io.

Timeline

Prize

Prizes will be awarded to for the top three submissions evaluated by RPSS and peer-review scores and must beat the re-calibrated ECMWF benchmark:

The 3rd prize is reserved for the top submission from developing or least developed country or small island state as per the UN list (see table C, F, H p.166ff). If such a submissions is already among the top 2, the third submission will get the 3rd prize.

Predictions

The organizers envisage two different approaches for Machine Learning-based predictions, which may be combined. Predict the week 3-4 & 5-6 state based on:

ML-based predictions schematic

For the exact valid_times to predict, see timings. For the data to use us for training, see data sources. Comply with the rules.

Evaluation

The objective of the competition is to improve week 3-4 (weeks 3 plus 4) and 5-6 (weeks 5 plus 6) subseasonal global probabilistic 2m temperature and total precipitation forecasts issued in the year 2020 by using Machine Learning/Artificial Intelligence.

The evaluation will be continuously performed by a scorer bot on renkulab.io, following the verification notebook. Submissions are evaluated on the Ranked Probability Score (RPS) between the ML-based forecasts and ground truth CPC temperature and accumulated precipitation observations based on pre-computed observations-based terciles. This RPS is compared to the re-calibrated real-time 2020 ECMWF forecasts into the Ranked Probability Skill Score (RPSS).

RPS is calculated with the open-source package xskillscore over all 2020 forecast_times. For deterministic forecasts:

xs.rps(observations, deterministic_forecasts, category_edges=precomputed_tercile_edges, dim='forecast_time')

For probabilistic forecasts:

xs.rps(observations, probabilistic_forecasts, category_edges=None, input_distributions='p', dim='forecast_time')

See the xskillscore.rps API for details.

def RPSS(rps_ML, rps_benchmark):
    """Ranked Probability Skill Score. Compares two RPS.
  
    +---------+-----------------------------------------+
    |  Score  | Description                             |
    +---------+-----------------------------------------+
    |    1    | maximum, perfect improvement            |
    +---------+-----------------------------------------+
    |  (0,1]  | positive means ML better than benchmark |
    +---------+-----------------------------------------+
    |    0    | equal performance                       |
    +---------+-----------------------------------------+
    | (0, -∞) | negative means ML worse than benchmark  |
    +---------+-----------------------------------------+

  """
  return 1 - rps_ML / rps_benchmark  # positive means ML better than ECMWF benchmark

The RPSS relevant for the prizes is first calculated on each grid cell over land globally on a 1.5 degree grid. This gridded RPSS is spatially averaged (weighted (np.cos(np.deg2rad(ds.latitude)))) over [90N-60S] land points and further averaged over both variables and both lead_times.

For diagnostics, we will further host leaderboards for the two variables in three regions:

Please find more details in the verification notebook (coming soon…).

Submissions

We expect submissions to cover all bi-weekly week 3-4 and week 5-6 forecasts issued in 2020, see timings. We expect one submission netcdf file for all 53 weekly forecasts issued in 2020. Submission must be gridded on a global 1.5 degree grid.

Each submission has to be a netcdf file with the folloing dimension sizes and coordinates:

>>> # in xarray
>>> ML_forecasts.sizes
Frozen(SortedKeysDict({'forecast_time': 53, 'latitude': 121, 'longitude': 240, 'lead_time': 2, 'category': 3}))

>>> ML_forecasts.coords
Coordinates:
  * latitude                 (latitude) float64 90.0 88.5 87.0 ... -88.5 -90.0
  * longitude                (longitude) float64 0.0 1.5 3.0 ... 357.0 358.5
  * forecast_time            (forecast_time) datetime64[ns] 2020-01...
  * lead_time                (lead_time) timedelta64[ns] 14 days 28 days
  * category                 (category) <U11 '[0., 0.33)' '[0.33, 0.66)' '[0.66, 1.]'
    valid_time               (lead_time, forecast_time) datetime64[ns] 2...

A template file for submissions will soon be available here.

The submissions have to be commited in git with git lfs in a repository hosted by renkulab.io.

After the competition, the code for training together with the gridded results must be made public, so the organizers and peer review can check adherence to the rules. Please indicate the resources used (number of CPUs/GPUs, memory, platform; see examples) in your scripts/notebooks to allow reproducibility and document them fully to enable easy interpretation of the codes. Submissions, which cannot be independently reproduced by the organizers after the competition ends, cannot win prizes, please see rules.

Data

Timings

The organizers explicitly choose to run this competition on past 2020 data, instead of a real-time competition to enable a much shorter competition period and to keep momentum high. We are aware of the dangers of overfitting (see rules), if the ground truth data is accessible.

Please find here an explicit list of the forecast dates required.

1) Which forecast starts/target periods (weeks 3-4 & 5-6) to require to be submitted?

Please find a list of the dates when forecasts are issued (forecast_time with CF standard_name forecast_reference_time) and corresponding start and end in valid_time for week 3-4 and week 5-6.

lead_time week 3-4 start week 3-4 end week 5-6 start week 5-6 end
forecast_reference_time
valid_time 2020-01-02 2020-01-16 2020-01-29 2020-01-30 2020-02-12
2020-01-09 2020-01-23 2020-02-05 2020-02-06 2020-02-19
2020-01-16 2020-01-30 2020-02-12 2020-02-13 2020-02-26
2020-01-23 2020-02-06 2020-02-19 2020-02-20 2020-03-04
2020-01-30 2020-02-13 2020-02-26 2020-02-27 2020-03-11
2020-02-06 2020-02-20 2020-03-04 2020-03-05 2020-03-18
2020-02-13 2020-02-27 2020-03-11 2020-03-12 2020-03-25
2020-02-20 2020-03-05 2020-03-18 2020-03-19 2020-04-01
2020-02-27 2020-03-12 2020-03-25 2020-03-26 2020-04-08
2020-03-05 2020-03-19 2020-04-01 2020-04-02 2020-04-15
2020-03-12 2020-03-26 2020-04-08 2020-04-09 2020-04-22
2020-03-19 2020-04-02 2020-04-15 2020-04-16 2020-04-29
2020-03-26 2020-04-09 2020-04-22 2020-04-23 2020-05-06
2020-04-02 2020-04-16 2020-04-29 2020-04-30 2020-05-13
2020-04-09 2020-04-23 2020-05-06 2020-05-07 2020-05-20
2020-04-16 2020-04-30 2020-05-13 2020-05-14 2020-05-27
2020-04-23 2020-05-07 2020-05-20 2020-05-21 2020-06-03
2020-04-30 2020-05-14 2020-05-27 2020-05-28 2020-06-10
2020-05-07 2020-05-21 2020-06-03 2020-06-04 2020-06-17
2020-05-14 2020-05-28 2020-06-10 2020-06-11 2020-06-24
2020-05-21 2020-06-04 2020-06-17 2020-06-18 2020-07-01
2020-05-28 2020-06-11 2020-06-24 2020-06-25 2020-07-08
2020-06-04 2020-06-18 2020-07-01 2020-07-02 2020-07-15
2020-06-11 2020-06-25 2020-07-08 2020-07-09 2020-07-22
2020-06-18 2020-07-02 2020-07-15 2020-07-16 2020-07-29
2020-06-25 2020-07-09 2020-07-22 2020-07-23 2020-08-05
2020-07-02 2020-07-16 2020-07-29 2020-07-30 2020-08-12
2020-07-09 2020-07-23 2020-08-05 2020-08-06 2020-08-19
2020-07-16 2020-07-30 2020-08-12 2020-08-13 2020-08-26
2020-07-23 2020-08-06 2020-08-19 2020-08-20 2020-09-02
2020-07-30 2020-08-13 2020-08-26 2020-08-27 2020-09-09
2020-08-06 2020-08-20 2020-09-02 2020-09-03 2020-09-16
2020-08-13 2020-08-27 2020-09-09 2020-09-10 2020-09-23
2020-08-20 2020-09-03 2020-09-16 2020-09-17 2020-09-30
2020-08-27 2020-09-10 2020-09-23 2020-09-24 2020-10-07
2020-09-03 2020-09-17 2020-09-30 2020-10-01 2020-10-14
2020-09-10 2020-09-24 2020-10-07 2020-10-08 2020-10-21
2020-09-17 2020-10-01 2020-10-14 2020-10-15 2020-10-28
2020-09-24 2020-10-08 2020-10-21 2020-10-22 2020-11-04
2020-10-01 2020-10-15 2020-10-28 2020-10-29 2020-11-11
2020-10-08 2020-10-22 2020-11-04 2020-11-05 2020-11-18
2020-10-15 2020-10-29 2020-11-11 2020-11-12 2020-11-25
2020-10-22 2020-11-05 2020-11-18 2020-11-19 2020-12-02
2020-10-29 2020-11-12 2020-11-25 2020-11-26 2020-12-09
2020-11-05 2020-11-19 2020-12-02 2020-12-03 2020-12-16
2020-11-12 2020-11-26 2020-12-09 2020-12-10 2020-12-23
2020-11-19 2020-12-03 2020-12-16 2020-12-17 2020-12-30
2020-11-26 2020-12-10 2020-12-23 2020-12-24 2021-01-06
2020-12-03 2020-12-17 2020-12-30 2020-12-31 2021-01-13
2020-12-10 2020-12-24 2021-01-06 2021-01-07 2021-01-20
2020-12-17 2020-12-31 2021-01-13 2021-01-14 2021-01-27
2020-12-24 2021-01-07 2021-01-20 2021-01-21 2021-02-03
2020-12-31 2021-01-14 2021-01-27 2021-01-28 2021-02-10

2) Which data to “allow” to be used to make a specific ML forecast?

Sources

Main datasets for this competition are already available as renku datasets and in climetlab for both variables temperature and total precipitation. In climetlab, we have one dataset lab for the Machine Learning community and S2S forecasting community, which both lead to the same datasets:

tag in climetlab (ML community) tag in climetlab (S2S community) Description renku dataset
training-output-reference observations-like-reforecasts CPC daily observations formatted as 2000-2019 reforecasts with forecast_time and lead_time missing
test-output-reference observations-like-forecasts CPC daily observations formatted as 2020 forecasts with forecast_time and lead_time missing
training-input hindcast-input daily real-time initialized on thursdays 2020 forecasts from models ECMWF, ECCC, NCEP missing
test-input forecast-input daily reforecasts initialized once per week until 2019 from models ECMWF, ECCC, NCEP missing
forecast-benchmark forecast-benchmark ECMWF week 3+4 & 5+6 re-calibrated real-time 2020 forecasts missing
tercile_edges tercile_edges Observations-based tercile category_edges missing

Note that tercile_edges separating observations into the category below normal [0.-0.33), normal [0.33-0.67) or above normal [0.67-1.] depend on longitude (240), latitude (121), lead_time (46 days or 2 bi-weekly), forecast_time.weekofyear (53) and category_edge (2), but are not yet available via climetlab yet.

We encourage to use subseasonal forecasts from the S2S and SubX projects:

However, any other publicly available data sources (like CMIP, NMME, etc.) of dates prior to the forecast_time can be used for training-input and forecast-input. Also purely empirical methods like persistence or climatology could be used. The only essential data requirement concerns forecast times and dates, see timings.

Ground truth sources are NOAA CPC temperature and total precipitation from IRIDL:

Examples

In progress…

Join

Follow the steps in the template renku project.

Training

Where to train?

How to train?

We are looking for your smart solutions here. Find a quick start template here soon.

Discussion

Please use the issue tracker in the renkulab s2s-ai-challenge gitlab repository for questions to the organizers, discussions, bug reports. We have set up a Gitter chat room for informal communication.

FAQ

Answered questions from the issue tracker are regularly transferred to the FAQ.

Leaderboard

RPSS

Submissions have to beat the ECMWF re-calibrated benchmark and following the rules to qualify for prizes.

The leaderboard will be made public after the submission period ends and submission codes have been made public, i.e. early November 2021.

We will also publish RPSS subleaderboards, that are purely diagnostic and show RPSS for two variables (t2m, tp), two lead_times (weeks 3-4 & 5-6) and three subregions ([90N-30N], (30N-30S), [30S-60S]).

Peer-review

From November 2021 to January 2022, there will be open peer review and expert peer-reviews for the top ranked submissions. Peer review will evaluate:

(Still under discussion) Based on these criteria, there will be a peer-review grade from poor (-1) towards brilliant (+1).

Final

The top three submissions based on the combined RPSS and expert peer-review score will receive prizes. (Still under discussion whether the scores are averaged or ranked and the two ranks averaged.)

Rules

Organizers