COSMO Priority Project ICON COMFORT:
ICON-COMpetence in FORecasTing

Last updated: 5 Dec 2022
get the pdf document

Project leader: Dimitra Boucouvala (HNMS)

0. Project resources 1. Goals 2. Motivation 3. Expected results 4. Tasks 5. Bibliography 6. FTEs & Timetable

Project resources

version:

24.02.2022

duration:

01/3/22-31/10/23

total FTEs:

1.36

Goals

The first goal of the project is the evaluation of ICON-LAM forecasts by forecasting departments and the translation of this feedback to the COSMO-ICON community. A second goal is to evaluate COSMO licensee NMHSs' satisfaction of the COSMO forecasts before the migration to ICON-LAM.

Motivation

The transition of the COSMO model to ICON-LAM is a new challenge for the forecasters who are asked to issue everyday forecasts and take decisions about warnings of extreme events. Apart from the operational verification procedures, the forecasters who receive on a daily basis the outputs of COSMO and ICON models, gain a subjective experience of the model performance according to the forecast outcome which can be based on different type of observations such as metars, radars or even public reports.

This performance of the models in different synoptic conditions and different seasons and regions after constant monitoring may detect model deficiencies therefore the forecasters' feedback can play an important role in model improvement. In addition to that, forecasters may have some special needs in terms of model outputs that can be useful for special types of forecasts such as TAF and sea route forecasts.

The above goal is achieved by providing forecasting departments and, if available, other COSMO and ICON-LAM users with regular surveys, preferably twice a year during this PP. The forecasters, according to the questions of the survey, may monitor the models performance, answer the survey at the end of each evaluation period, and collect some cases which will be representative of the following:

Furthermore, additional requirements of ICON-LAM users with respect to data format and output meteorological variables will be reported by the forecasters and will be taken into account for postprocessing software development (not part of this project).

figure 1
diagram of PP workflow. Interaction between forecasters-modelers through evaluation tasks.

Expected results

A report including analysis of forecasters' feedback of subjective evaluation of ICON-LAM performance in respect to COSMO forecasts by means of regular surveys and representative test cases, in addition to the comparison with objective verification outcome. The main findings on the evaluation of ICON-D2 (-EPS) products at the ESSL (European Severe Storms Laboratory) Testbed for 2020 and 2021 will also be considered.

The IC–ěN-COMFORT PP might be considered as a pilot for a continuous COSMO activity in evaluating the forecasts by forecasters after the completion of the project , e.g. within the SPRT activity, with some FTEs, e.g., 0.05 FTE per year for each participating institute, and 0.2 for a person/persons preparing the overview and comparison with objective verification results.

After the transition from COSMO to ICON, the lessons learned from the PP ICON-COMFORT can be a starting point of such a permanent yearly activity aiming to ICON-LAM forecasts subjective evaluation, with the perspective of model improvement. To this end, the survey will be updated given that most of the participants should complete the transition to ICON by the time the PP ICON-COMFORT finishes.

For the COSMO licencees countries, the survey will help to analyze the need for the support activities needed for the transition from COSMO to ICON-LAM.

Tasks

Task 1: Survey completion and analysis

Surveys will be sent to a WG4 representative from each of the participating institutions to the PP. This representative, will be responsible for distributing it to the forecasters, providing the answers to the surveys, collect representative cases, and write a report including the general outcomes of the subjective evaluation from the forecasters. The evaluation period of the survey is one year and the survey completion and eventual update are twice a year. Each participant can focus his feedback both on the survey summary as well as concentrate on representative events during the evaluation period.

Total FTEs for Task 1: 0.76

Sub-Task 1.1: Survey processing

Survey update by the PP leader, management of the activity (PP leader), distribution and collection of answers from the forecasters from the representative in each service who may select representative weather events to be further evaluated. The sub-task 1.1 will be repeated twice a year.

Participating scientists:

Dimitra Boucouvala (HNMS) (0.03+0.04 FTE)
Josué Gehring (MCH) (0.03 FTE)
Alessio Canessa (CNMCA) (0.03 FTE)
Ramona Dragomir (NMA) (0.03 FTE)
Anastasia Bundel (RHM) (0.03 FTE)
Eugeni Brainin (IMS) (0.03 FTE)
Valeria Garbero (ARPA-P) (0.03 FTE)
Joanna Linkowska (IMGW) (0.03 FTE)

FTEs:

0.28

Sub-Task 1.2: Analysis of Surveys and ICON-LAM forecast skill on specific weather events

Report including forecasters' feedback with outcomes of Task 1.1 and analysis of the representative cases during the forecasting period.

 

Deliverables:

Report from each participant with summarized answers on the survey accompanied by the evaluation analysis for ICON performance. Content of the deliverable report is in line with Task 1.1 description. The deliverables of task 1.2 are twice a year.

Participating scientists:

Dimitra Boucouvala, Ioannis Kouroutzoglou (HNMS) (0.06 FTE)
Josué Gehring (MCH) (0.06 FTE)
Alessio Canessa (CNMCA) (0.06 FTE)
Ramona Dragomir (NMA) (0.06 FTE)
Anastasia Bundel (RHM) (0.06 FTE)
Eugeni Brainin (IMS) (0.06 FTE)
Valeria Garbero (ARPA-P) (0.06 FTE)
Joanna Linkowska (IMGW) (0.06 FTE)

FTEs:

0.48

Task 2: Evaluation of ICON-LAM forecast skill

Deliverables:

Final report containing the main findings extracted from forecasters' feedback of all services, with additional information on requirements on forecast products and representation tools and comparison with objective verification. The report will contain the section about the lessons learnt during the task including organizational issues (desirable and feasible frequency of survey distribution, interrelationships with different departments, etc.).

Critical Issues

For the departments that will be using data assimilation, it is reasonable to provide the forecasting departments with the ICON-LAM forecasts and the survey only after a data assimilation procedure is established. Retrieving forecasters' feedback before establishing a data assimilation procedure is voluntary and should only be performed in case internal verification showed an added value compared to COSMO-model forecasts.

Total FTEs for Task 2: 0.25

Sub-Task 2.1: Forecasters Feedback assessment

Using the outcome of Task 1 deliverables, a final assessment will be performed with the evaluation of the subjective verification by forecasters of all services. Through this, the overall operational skill of ICON-LAM models will be described, while systematic errors and additional requirements on forecast products and representation tools will be provided.

Participants:

D. Boucouvala (HNMS) (0.1 FTE)
A. Bundel (RHM) (0.1 FTE)

FTEs:

0.2

Sub-Task 2.2: Systematic errors vs Forecasters Feedback

The objective verifications obtained on a seasonal basis through by WG5 Common Plot activity, will be also compared with the outcome of Task 2.1 in an effort to detect differences or similarities among these two approaches for the seasonal model error tendencies.

Participants:

D. Boucouvala (HNMS) (0.05 FTE)

FTEs:

0.05

Task 3: Evaluation of COSMO licencees satisfaction

The COSMO model has been used operationally under a licence agreement by the national meteorological services of the United Arab Emirates (NCMS), Brazil (INMET), Oman (DGMAN), Botswana (BMS), Brazilian Navy (DHN), and Turkmenistan (Turkmen Gidromet). The annual licence fee for the operational usage of the COSMO model is 20,000 € per service but there is no charge for least developed countries. This allows e.g. Burkina Faso, Egypt, Georgia, Indonesia, Kenya, Madagascar, Malawi, Mozambique, Nigeria, Pakistan, Philippines, Rwanda, Sudan, Tanzania, Ukraine, Vietnam, and Zimbabwe to use the COSMO model free of charge. Besides the COSMO Licencees already running the COSMO model in daily production, a few weather services are evaluating the usage of our model system. Among them are the national weather services of Armenia, Azerbaijan, and South Africa. They currently have a scientific licence.

The funds acquired from licence fees are used to promote collaboration between COSMO partners and for training activities, with the aim to enhance the model performance. The COSMO spirit was built during the years by many meetings which were partly funded by the licencees countries.

Although it is still under negotiations it seems that ICON will be an open-source code and the payment to COSMO will be for support activities. Furthermore, COSMO can give COSMO/ICON users a sense of community and an added value that will be worthwhile to pay.

A first step in the direction of establishing a community and treating these countries as our customers will be a short survey to evaluate their current satisfaction from COSMO forecasts and the support activities.

Sub-Task 3.1: Survey creation and distribution to COSMO users

Create with Google forms a survey that will include subjects as:

Sub-Task 3.2: Survey collection and analysis

Surveys will be collected by the Google forms tool. The results will be analyzed, together with the model performance documentation.

Deliverables:

A report which will include the feedback analyses and suggestions to improve the support. The conclusion will also deal with the plan for migration from COSMO to ICON-LAM. The outcomes will be presented in the COSMO general meeting.

Participating scientists:

Dimitra Boucouvala (HNMS) (0.08 FTE)
Rodica Dumitrache (NMA) (0.08 FTE)
Bogdan Maco (NMA) (0.05 FTE)
Alon Shtivelman (0.05 FTE)
Daniel Rieger (DWD) (0.03 FTE)
Yoav Levi (IMS) (0.02 FTE)

Bibliography

FTEs & Timetable

1. Analysis of Resources
Task Contributors FTEs Deliverables Date of
delivery
SON 2022 DJF 2022 MAM 2023 JJA 2023
1.1 Updating Survey- Management D. Boucouvala (HNMS) 0.02   0.02   Update forecaster Survey Each evaluation period
Distribution at MCH J. Gehring 0.015   0.015     Each evaluation period
Distribution at CNMCA A. Canessa 0.015   0.015     Each evaluation period
Distribution at HNMS D.Boucouvala 0.015   0.015     Each evaluation period
Distribution at IMGW J. Linkowska 0.015   0.015     Each evaluation period
Distribution at NMA R. Dragomir 0.015   0.015     Each evaluation period
Distribution at RHM A. Bundel 0.015   0.015     Each evaluation period
Distribution at IMS E. Brainin 0.015   0.015     Each evaluation period
Distribution at ARPA-P. V. Garbero 0.015   0.015     Each evaluation period
1.2 Evaluation at MCH J.Gehring   0.03   0.03 Report 03.2023
09.2023
Evaluation at CNMCA A. Canessa   0.03   0.03 Report 03.2023
09.2023
Evaluation at HNMS D.Boucouvala
I.Kouroutzoglou
  0.03   0.03 Report 03.2023
09.2023
Evaluation at IMGW J. Linkowska   0.03   0.03 Report 03.2023
09.2023
Evaluation at NMA R. Dragomir   0.03   0.03 Report 03.2023
09.2023
Evaluation at RHM A. Bundel   0.03   0.03 Report 03.2023
09.2023
Evaluation at IMS E. Brainin   0.03   0.03 Report 03.2023
09.2023
Evaluation at ARPA-PT V. Garbero   0.03   0.03 Report 03.2023
09.2023
2.1 Report on subj. evaluation D. Boucouvala
A. Bundel
      0.1
0.1
Final Report 10.2023
2.2 Comparison Subj./Obj eval D. Boucouvala       0.05 Contribution to Report (2.1) 10.2023
      MAM 2022 JJA 2022 MAM 2023 JJA 2023    
3.1 writing the survey all 0.17       Google forms survey 6.2022
3.2 survey analysis all   0.1   0.1 report 10.2023

 

 

2. Timetable
date/
task
03/22 08/22 09/22 02/23 03/23 03/23 08/23 09/23 10/23
1.1     start end   start end    
1.2       start end   start end  
2.1         start       end
2.2         start       end
3.1 start end              
3.2   start             end