Project resources

Goals

The first goal of the project is the evaluation of ICON-LAM forecasts by forecasting departments and the translation of this feedback to the COSMO-ICON community. A second goal is to evaluate COSMO licensee NMHSs' satisfaction of the COSMO forecasts before the migration to ICON-LAM.

Motivation

The transition of the COSMO model to ICON-LAM is a new challenge for the forecasters who are asked to issue everyday forecasts and take decisions about warnings of extreme events. Apart from the operational verification procedures, the forecasters who receive on a daily basis the outputs of COSMO and ICON models, gain a subjective experience of the model performance according to the forecast outcome which can be based on different type of observations such as metars, radars or even public reports.

This performance of the models in different synoptic conditions and different seasons and regions after constant monitoring may detect model deficiencies therefore the forecasters' feedback can play an important role in model improvement. In addition to that, forecasters may have some special needs in terms of model outputs that can be useful for special types of forecasts such as TAF and sea route forecasts.

The above goal is achieved by providing forecasting departments and, if available, other COSMO and ICON-LAM users with regular surveys, preferably twice a year during this PP. The forecasters, according to the questions of the survey, may monitor the models performance, answer the survey at the end of each evaluation period, and collect some cases which will be representative of the following:

• added value of ICON-LAM compared to the COSMO-model for different weather regimes,
• identification of ICON-LAM decreased model skill for certain regions or certain weather regimes,
• added value of ICON-LAM compared to the COSMO-model in case of severe weather situations.
• persistent biases in forecasted parameters that have not been corrected with the newer modelling systems (model, version)

Furthermore, additional requirements of ICON-LAM users with respect to data format and output meteorological variables will be reported by the forecasters and will be taken into account for postprocessing software development (not part of this project).

Expected results

A report including analysis of forecasters' feedback of subjective evaluation of ICON-LAM performance in respect to COSMO forecasts by means of regular surveys and representative test cases, in addition to the comparison with objective verification outcome. The main findings on the evaluation of ICON-D2 (-EPS) products at the ESSL (European Severe Storms Laboratory) Testbed for 2020 and 2021 will also be considered.

The ICОN-COMFORT PP might be considered as a pilot for a continuous COSMO activity in evaluating the forecasts by forecasters after the completion of the project , e.g. within the SPRT activity, with some FTEs, e.g., 0.05 FTE per year for each participating institute, and 0.2 for a person/persons preparing the overview and comparison with objective verification results.

After the transition from COSMO to ICON, the lessons learned from the PP ICON-COMFORT can be a starting point of such a permanent yearly activity aiming to ICON-LAM forecasts subjective evaluation, with the perspective of model improvement. To this end, the survey will be updated given that most of the participants should complete the transition to ICON by the time the PP ICON-COMFORT finishes.

For the COSMO licencees countries, the survey will help to analyze the need for the support activities needed for the transition from COSMO to ICON-LAM.

Tasks

Task 1: Survey completion and analysis

Surveys will be sent to a WG4 representative from each of the participating institutions to the PP. This representative, will be responsible for distributing it to the forecasters, providing the answers to the surveys, collect representative cases, and write a report including the general outcomes of the subjective evaluation from the forecasters. The evaluation period of the survey is one year and the survey completion and eventual update are twice a year. Each participant can focus his feedback both on the survey summary as well as concentrate on representative events during the evaluation period.

Total FTEs for Task 1: 0.76

Sub-Task 1.1: Survey processing

Survey update by the PP leader, management of the activity (PP leader), distribution and collection of answers from the forecasters from the representative in each service who may select representative weather events to be further evaluated. The sub-task 1.1 will be repeated twice a year.

Sub-Task 1.2: Analysis of Surveys and ICON-LAM forecast skill on specific weather events

Report including forecasters' feedback with outcomes of Task 1.1 and analysis of the representative cases during the forecasting period.

 

Task 2: Evaluation of ICON-LAM forecast skill

Sub-Task 2.1: Forecasters Feedback assessment

Using the outcome of Task 1 deliverables, a final assessment will be performed with the evaluation of the subjective verification by forecasters of all services. Through this, the overall operational skill of ICON-LAM models will be described, while systematic errors and additional requirements on forecast products and representation tools will be provided.

Sub-Task 2.2: Systematic errors vs Forecasters Feedback

The objective verifications obtained on a seasonal basis through by WG5 Common Plot activity, will be also compared with the outcome of Task 2.1 in an effort to detect differences or similarities among these two approaches for the seasonal model error tendencies.

Task 3: Evaluation of COSMO licencees satisfaction

The COSMO model has been used operationally under a licence agreement by the national meteorological services of the United Arab Emirates (NCMS), Brazil (INMET), Oman (DGMAN), Botswana (BMS), Brazilian Navy (DHN), and Turkmenistan (Turkmen Gidromet). The annual licence fee for the operational usage of the COSMO model is 20,000 € per service but there is no charge for least developed countries. This allows e.g. Burkina Faso, Egypt, Georgia, Indonesia, Kenya, Madagascar, Malawi, Mozambique, Nigeria, Pakistan, Philippines, Rwanda, Sudan, Tanzania, Ukraine, Vietnam, and Zimbabwe to use the COSMO model free of charge. Besides the COSMO Licencees already running the COSMO model in daily production, a few weather services are evaluating the usage of our model system. Among them are the national weather services of Armenia, Azerbaijan, and South Africa. They currently have a scientific licence.

The funds acquired from licence fees are used to promote collaboration between COSMO partners and for training activities, with the aim to enhance the model performance. The COSMO spirit was built during the years by many meetings which were partly funded by the licencees countries.

Although it is still under negotiations it seems that ICON will be an open-source code and the payment to COSMO will be for support activities. Furthermore, COSMO can give COSMO/ICON users a sense of community and an added value that will be worthwhile to pay.

A first step in the direction of establishing a community and treating these countries as our customers will be a short survey to evaluate their current satisfaction from COSMO forecasts and the support activities.

Sub-Task 3.1: Survey creation and distribution to COSMO users

Create with Google forms a survey that will include subjects as:

• Information on the current COSMO status
• The last interaction with the COSMO support team
• The last interaction with DWD support for boundary conditions
• The satisfaction with the support activities
• The satisfaction with the COSMO forecasts.
• Available COSMO visualization tools and post-processing needs.
• Availability of verification products.
• Availability of an expert for participation on a training course on ICON-LAM at DWD.
• Availability of an expert for installing and running ICON-LAM
• Supplementary documents of model performance (can be in the local language)

Sub-Task 3.2: Survey collection and analysis

Surveys will be collected by the Google forms tool. The results will be analyzed, together with the model performance documentation.

Bibliography

• COSMO Documentation
• WG5 CP Verification
• PP C2I
• COSMO Licencing

FTEs & Timetable

1. Analysis of Resources

Task		Contributors	FTEs				Deliverables	Date of delivery
			SON 2022	DJF 2022	MAM 2023	JJA 2023
1.1	Updating Survey- Management	D. Boucouvala (HNMS)	0.02		0.02		Update forecaster Survey	Each evaluation period
	Distribution at MCH	J. Gehring	0.015		0.015			Each evaluation period
	Distribution at CNMCA	A. Canessa	0.015		0.015			Each evaluation period
	Distribution at HNMS	D.Boucouvala	0.015		0.015			Each evaluation period
	Distribution at IMGW	J. Linkowska	0.015		0.015			Each evaluation period
	Distribution at NMA	R. Dragomir	0.015		0.015			Each evaluation period
	Distribution at RHM	A. Bundel	0.015		0.015			Each evaluation period
	Distribution at IMS	E. Brainin	0.015		0.015			Each evaluation period
	Distribution at ARPA-P.	V. Garbero	0.015		0.015			Each evaluation period
1.2	Evaluation at MCH	J.Gehring		0.03		0.03	Report	03.2023 09.2023
	Evaluation at CNMCA	A. Canessa		0.03		0.03	Report	03.2023 09.2023
	Evaluation at HNMS	D.Boucouvala I.Kouroutzoglou		0.03		0.03	Report	03.2023 09.2023
	Evaluation at IMGW	J. Linkowska		0.03		0.03	Report	03.2023 09.2023
	Evaluation at NMA	R. Dragomir		0.03		0.03	Report	03.2023 09.2023
	Evaluation at RHM	A. Bundel		0.03		0.03	Report	03.2023 09.2023
	Evaluation at IMS	E. Brainin		0.03		0.03	Report	03.2023 09.2023
	Evaluation at ARPA-PT	V. Garbero		0.03		0.03	Report	03.2023 09.2023
2.1	Report on subj. evaluation	D. Boucouvala A. Bundel				0.1 0.1	Final Report	10.2023
2.2	Comparison Subj./Obj eval	D. Boucouvala				0.05	Contribution to Report (2.1)	10.2023
			MAM 2022	JJA 2022	MAM 2023	JJA 2023
3.1	writing the survey	all	0.17				Google forms survey	6.2022
3.2	survey analysis	all		0.1		0.1	report	10.2023

2. Timetable

date/ task	03/22	08/22	09/22	02/23	03/23	03/23	08/23	09/23	10/23
1.1			start	end		start	end
1.2				start	end		start	end
2.1					start				end
2.2					start				end
3.1	start	end
3.2		start							end