KENDAScope: KENDA from Surface to Cloud Observations Progressive Extension

Last updated: 27 May 2021
get the pdf document

Project leader: Christoph Schraff (DWD)

0. Project resources 1. Summary 2. Background and motivation 3. Actions proposed 4. Risks 5. Links with other projects or activities 6. Description of individual tasks 7. References 8. Appendix: Task table

0. Project resources

Version

0.3, 10 July 2020

Project duration:

Sept. 2020-Aug. 2025 (5 years)

Estimated resources for 2021-2025:

42 FTE

Estimated resources in 2021:

12 FTE

1. Summary

The aim of the project is to further improve and extend the KENDA data assimilation system and use of observations in view of better convective-scale deterministic and ensemble forecasts with ICON-LAM particularly for weather-related quantities. On the one hand, this is addressed by extending the use of observations, with a focus on those related to cloud and precipitation, but also those in the planetary boundary layer and near the surface.

On the other hand, the current LETKF scheme will be complemented by a variational component for an ensemble-variational analysis (EnVar) and, at least experimentally, by a particle filter (PF) in order to address certain limitations of the LETKF. Furthermore, tools will be provided to perform a variational data assimilation without the need to run a (convective-scale) ensemble in the assimilation cycle. This offers the possibility to carry out data assimilation for ICON-LAM at similar low cost as it was possible with the observation nudging scheme for the COSMO model.

2. Background and motivation

In the precedent projects KENDA (Km-scale ENsemble-based Data Assimilation) and KENDA-O (KENDA for high-resolution Observations), a 4-dimensional Local Ensemble Transform Kalman Filter (4D-LETKF) scheme following Hunt et al. (2007) has been developed for the COSMO model and ported to the ICON-LAM model. Latent heat nudging (LHN, Stephan et al., 2008) of radar precipitation has been integrated into the KENDA-LETKF analysis cycle.

The main purpose of this data assimilation (DA) system called KENDA (Schraff et al., 2016) is to provide the initial conditions both for deterministic and ensemble forecasting on the convective scale (i.e. with 1-3 km model mesh size). Foci of interest include improved forecasts of high-impact weather, e.g. in convective situations or for fog and low stratus conditions, and especially in the very short time range. It is running operationally now at several COSMO member states (Germany, Switzerland, Italy), providing the initial conditions for operational forecasts.

Compared to the previous observation nudging scheme, the introduction of KENDA has resulted overall in large improvements in the EPS forecasts and more moderate but still significant improvements in the deterministic forecasts, particularly of convective precipitation in summer. This benefit of KENDA prevails throughout except for two aspects:a) low stratus, which has been degraded in some cases, notably over the Swiss Plateau, and b) 2-m humidity which has been found to be slightly degraded (in MeteoSwiss verifications) since 2-m humidity observations have been assimilated only in the nudging scheme up to now. Apart from the 2-m humidity data, these overall very positive results were achieved using a similar set of observations.

The ability of the KENDA-LETKF to derive and use flow-dependent background error covariances for the computation of the analysis increments certainly plays a major role for this. Compared to the nudging scheme with its limitations to assimilate indirect observations, the LETKF also offers much better prospects to use additional data. The greatest success so far has been the recent operational introduction of radar radial velocity and reflectivity volume data in the LETKF, with clear benefits found in the main tests (at DWD and at ARPAE (Gastaldo et al., 2018)). With this, the COSMO consortium is the first one (in Europe and at national level worldwide to our knowledge) to assimilate operationally 3-D radar reflectivity directly without any retrieval approach.

Besides this great potential of the LETKF, there are also certain limitations and issues (some of which may contribute to the shortcomings with low stratus), e.g.:

Other data assimilation methods offer advantages for some of these issues but also have their own disadvantages compared to the LETKF:

The particle filter (PF) addresses particularly non-Gaussianity and nonlinearity. At DWD, two PF variants, a Localized Adaptive Particle Filter (LAPF, see Potthast et al., 2019) and a Localized Markov Chain Particle Filter (LMCPF), have been developed. They compute an ensemble transform matrix locally as in the LETKF, but replace the LETKF update in ensemble space by a PF update and resampling in ensemble space. In this way, they overcome the problems of filter collapse (i.e. having far too little spread) and divergence (from the true state) that are typical for traditional PF applied to high-dimensional problems. Potthast et al. (2019) were the first to run a PF stably over a long period for a global NWP system in a quasi-operational setting and obtained forecast quality comparable to that from the LETKF. The LMCPF has also been found to work similarly for the COSMO model on the convective scale.

Compared to global DA, non-linearity and non-Gaussianity tend to play a larger role for convective-scale DA, both with respect to the involved (convective) model processes with their short time scales and with respect to the higher priority to assimilate data related to cloud and precipitation with highly nonlinear observation operators. Therefore, it appears particularly promising to continue the research on PF in the context of ICON-LAM. This will also attain a lot of attention and visibility in the scientific community.

In variational data assimilation (Var), certain aspects of non-linearity and non-Gaussianity can also be better handled than in the LETKF. The cost function may include nonlinear terms, and variational quality control (VarQC) helps to deal with observation departure statistics with fat tails by weighting down outliers. The iterative approach can improve the analysis in the presence of a non-quadratic shape of the cost function; on the other hand, the tangent linear and adjoint are required for the observation operators and in 4DVar for the NWP model. Furthermore, Var does not suffer from the rank deficiency problem and may potentially allow for using data at higher density (depending on the background error covariances applied), and it avoids the localisation in observation space.

At DWD, an EnVar scheme following ideas from Buehner et al. (2005) has been developed and is running operationally for global DA/NWP. Technically, it can be seen as a natural evolution of the 3DVar scheme combining the climatological background error covariances (B-matrix) of 3DVar (currently with a weight of 30 %) with the B-matrix derived from the ensemble perturbations of the associated LETKF (with a weight of 70 % as obtained by tuning). In this way, this hybrid EnVar (‘hybrid’ commonly denotes this type of blending of B-matrices) combines the main advantages of the two methods, Var and LETKF:It avoids or mitigates the above-mentioned shortcomings of the LETKF and at the same time benefits from the flow-dependent ensemble B-matrix. As a result, the forecasts started from EnVar analyses have been found to be significantly superior to those from the former 3DVar and to those from the analysis mean of the LETKF (which however is run at three times lower resolution). A four-dimensional version of EnVar known as 4D-EnVar is under development at DWD, details will be discussed below..

Note that EnVar provides only a single analysis for a deterministic forecast (unless a whole ensemble of EnVar’s is run) and still relies on the LETKF for the ensemble initial conditions. However, it is possible to apply ‘re-centering’ of the analysis ensemble around the EnVar deterministic analysis (by shifting all ensemble members in the model space in the same way such that the mean of the re-centered analysis ensemble coincides with the deterministic analysis). Recent tests for the global ICON system have shown large improvements of the EPS forecasts, when re-centering was applied to the LETKF ensemble during the data assimilation cycle and to the final analysis ensemble. Small improvements are obtained even in the deterministic analysis and forecast which obviously benefit from the improved B-matrix derived from the re-centered ensemble.

These results and considerations strongly promote the development and test of the EnVar for the convective scale (ICON-LAM) even though EnVar produces only a deterministic analysis and the focus of the COSMO consortium is on convective-scale EPS which would still rely on the (possibly re-centered) LETKF. For DWD, using EnVar operationally for ICON-LAM would also further harmonize the global and regional DA systems and increase synergies in research, development, and maintenance. It is worth mentioning that this development moves in parallel to the major trend of methods used at other major centres or European consortia for convective-scale DA:The UK Met Office deploy 1-hourly 4DVar (currently with climatological B-matrix, but working towards a hybrid B-matrix, see Milan et al., 2020). MeteoFrance, HIRLAM (HARMONIE-AROME), and JMA all use 3DVar but are working towards 4DVar and testing hybrid EnVar and 4D-EnVar (Gustafsson et al., 2018). NOAA-NCEP plan to introduce hybrid EnVar (HRRRDAS) in late autumn 2020.

Despite DWD’s encouraging experience with EnVar and recentering of the ensemble, it has yet to be seen, however, whether EnVar will really be able to improve upon the current KENDA-LETKF, for several reasons:

Yet, there is another important reason for the COSMO consortium to develop and provide Var. For the COSMO model, observation nudging has allowed institutes from member states or licensees with quite limited computational resources to carry out data assimilation as the nudging scheme is cheap and does not require to run a (convective-scale) ensemble in the data assimilation cycle in contrast to LETKF. As observation nudging is not available for ICON, a cheap surrogate approach without the need to run a convective-scale ensemble is needed, and this can be provided by Var.

Besides developing these algorithmic aspects, it is also very important to further extend the set of assimilated observation types for operational purposes. By and large, this will continue the research and development carried out in the previous KENDA-O project. Besides Mode-S aircraft data, radar radial velocity and reflectivity (for earlier studies, see Bick et al., 2016; Gastaldo et al., 2018) could be introduced operationally and successfully into KENDA within the framework of that project. It is now important to carry on maintaining, refining, improving, and (geographically) extending the use of these data and possibly adjusting it to significant changes in the model (e.g. 2-moment microphysics) and in the assimilation scheme as mentioned above. For some other observation types such as GNSS (Global Navigation Satellite System) zenith and slant total delay (ZTD and STD) and 2-m temperature and humidity from Synop stations (used to modify the atmospheric state), promising results have already been obtained, and it appears to require only few development steps and more testing to allow for operational use. There are also observation types such as all-sky satellite radiances where work is ongoing and more work is needed despite the progress achieved.

Even though any type of observation can be beneficial and important to be used, there are two classes of data for which further research, development, and extended use are particularly relevant for convective-scale analysis and forecasting of weather, in particular locally driven (or influenced) high-impact weather such as deep convection or low stratus and fog. The first class is high-resolution data related to humidity, cloud, and precipitation. Besides radar reflectivity (precipitation) and ZTD/STD (humidity) as mentioned above, a focus is on the all-sky VIS and IR satellite radiances (here from Meteosat) in view of assimilation of information on cloudiness (for an overview on all-sky satellite DA, see Geer et al., 2018; Kurzrock et al., 2018). For VIS radiances, a novel efficient forward operator (MFASIS) has been built, and promising results have already been obtained in a comprehensive case study for COSMO-KENDA by Scheck et al. (2020) and even for longer test periods with ICON-LAM at DWD. Related to the assimilation of IR water vapour (WV) radiances, Hutt et al. (2020) assimilated clear-sky WV radiances in COSMO-KENDA, while for cloudy (all-sky) IR WV radiances, Harnisch et al. (2016) introduced a model for observation errors, and Otkin and Potthast (2019) introduced a conditional nonlinear bias correction. Even though these are important steps towards the assimilation all-sky IR radiances in the operational setting of KENDA this topic will require more efforts.

The second class of observations that appear particularly relevant are observations describing the planetary boundary layer (PBL), down to the surface level. These include ground-based remote sensing devices such as microwave radiometers (MWR:for low-resolution temperature and humidity profiles), wind lidars, Raman lidars (temperature and humidity profiles), DIAL lidars (humidity) but also direct measurements for drones and towers. Some of these devices are still rare and experimental, while others are increasingly available. Leuenberger et al. (2020) have shown that high-frequent profile observations from Raman lidars and drones can improve analyses of the pre-convective boundary layer and the stable boundary layer favourable for fog and forecasts for cloudiness and precipitation up to +9h into forecast time. There are national projects dealing with several of these data types which need to be coordinated and supported as part of the further KENDA development. For the above-mentioned screen-level observations (2-m temperature, 2-m humidity), a novel approach for station-dependent bias correction has been devised which can also be deployed for observation error specification. It has great promise to be able to deal with networks with very variable quality. Therefore, it offers also a potential to obtain a positive impact from data from other observing networks than operated by national weather services, e.g. roadside sensors. These methods may also be further refined with AI and machine learning algorithms.

Another future data source considered highly important due to its vast amount and high spatial and temporal resolution is the MTG IRS (Meteosat Third Generation Infrared Sounder) which is scheduled to be operationally available in 2024. Preparations for the use of these data should possibly start soon.

Finally, some analysis tools related to the lower boundary conditions need to be consolidated and refined, including the soil moisture analysis using soil moisture data. An important step will be their port to resp. re-write in the framework of the DACE code which will make their maintenance and further development (including upgrades to more advanced analysis methods) much more efficient. This refers to the sea surface temperature (SST) analysis, the snow analysis, the soil moisture analysis and the related 2-m temperature analysis. Depending on the available resources at the later stages of the project, more efforts should go into a coupled data assimilation of the soil (moisture and possibly temperature) and atmosphere.

3. Actions proposed

  1. Algorithmic developments:Maintain and further refine the KENDA-LETKF; develop EnVar for operational purposes for ICON-LAM and compare with LETKF; further develop, refine, and test 4D-EnVar and PF; provide CEnVar and the tools for 3DVar as a means to carry out data assimilation for ICON-LAM without the need to run an ensemble (data assimilation) at the convective scale.
  2. Extend the use of observations in the KENDA system, in particular those related to the PBL, clouds and precipitation. Also prepare the use of MTG IRS data.
  3. Consolidate/redevelop analyses related to the surface:SST analysis, soil moisture analysis. The coordination related to and further development of the snow analysis will take place in a dedicated priority project, outside KENDAscope.

Even though some of the development work (outside DWD) in actions 2) and 3) may be carried out still with the COSMO model in the first years of the project, KENDAscope will be clearly focussing on the further development of KENDA for ICON-LAM.

4. Risks

KENDAscope does not consist of one main single task or aim which can be fulfilled or not. Therefore, the risk of an overall failure is low. Instead, there is a variety of individual tasks, each one dealing with its own scientific issues (many of which are mentioned above or in the respective tasks) and risks. For instance, there is a general risk that the gain in forecast quality from the use of each of the observation types addressed in KENDAscope judged on a larger scale may be difficult to prove and does not meet the expectations. As mentioned above there is e.g. also a risk that EnVar does not perform better than LETKF, in particular if observations related to cloud and precipitation, such as radar reflectivity or VIS radiances, are used. It is noted that even though operational applicability of the related development is the aim (deliverable) in many tasks, this is not always the case. For some of the topics, the work will very likely continue after KENDAscope in order to achieve further improvements.

The human resources appear to be secured in the earlier phases of the project for almost all actions (tasks). However, this is not always the case for the later stages of the 5-year period. Even though the past experience with the former KENDA-related 5-year projects makes us confident to be able to (re-)allocate the envisaged or required resources, this may not always be the case, particularly in case of severe financial cut-downs e.g. as a consequence of the pandemic. For tasks relying on national projects and hence on non-permanent positions, there is always a risk that the scientists leave the project early and cannot be replaced (adequately). Furthermore, unexpected urgent matters with highest priority (e.g. maternity leaves, operational duties, etc.) may detract planned resources and lead to delays (Task 1 may be exposed most to this risk). From today’s point of view, the largest risks for failing to make available the required FTE’s that are not yet allocated appear to be for adapting the direct assimilation of radar reflectivity to EnVar, for the particle filter, and for the soil moisture analysis using satellite soil moisture data.

5. Links with other projects or activities

COSMO WG7/PP PROPHECY:
COSMO WG3a:

Apart from the general importance to reduce model biases for the data assimilation, there are some strong direct links:

COSMO WG3b:

A separate priority project on snow analysis will be set up soon by WG3b together with WG1, see further comments in KENDAscope Task 3.

COSMO WG5:

COSMO WG5:The observation-based verification against single (non-gridded) observations mostly relies on the MEC (Model Equivalent Calculator) which has been developed and is maintained in KENDA priority projects (under the umbrella of the task on the further development of the LETKF). Requests from WG5 e.g. to monitor additional observation types will make it necessary to implement additional code types and observation operators in the KENDA code.

SINFONY (Seamless Integrated ForecastiNg sYstem):

This is a large project at DWD for the development of a coupled probabilistic system of precipitation nowcasting and very short-range NWP. Some FTE’s in KENDAscope are funded in the context SINFONY; topics include more realistic model physics (e.g. 2-moment microphysics), a stronger focus on the first few hours of the forecasts and therefore on use of high-resolution data, assimilation of (nowcast) objects, etc.

ICamCloudOps:

Joint project of DWD and the private company Reuniwatt; this includes 2 positions (Q2/2020-Q2/2024) at DWD on intelligent observation operators for cloud information from conventional and infrared cameras, using artificial intelligence/machine learning (link to KENDAscope Task 2.3).

SPP PROM (Polarimetric Radar Observations meet atmospheric Modelling) programme funded by DFG:

This includes a 3-year project on the development of forward operators for polarimetric parameters and inclusion in EMVORADO (which is used in the KENDA system for the assimilation of radar reflectivity and radial velocity) (link to KENDAscope Task 2.1).

RealPEP (Near Realtime Quantitative Precipitation Estimation and Prediction) ‘Research Group’ funded by DFG:

This includes 3-year sub-projects on the assimilation of radar polarimetric information and observation-based nowcasted fields (link (mainly) to KENDAscope Task 2.1).

Hans-Ertel Centre (HErZ)/LMU University of Munich:
IAFE GPU (4-year project at DWD starting Q1/2021):

The main aim is to explore and understand which parts of the DACE data assimilation code are feasible to be ported to GPU and how. This is in order to prepare the operational usability of DACE on new computer architectures and therefore very important for the future applicability of the whole KENDA system and developments. However, it is not the task of this project to perform the whole porting. For this, more resources will likely be required, and if other COSMO members will plan to contribute to this, a dedicated PP outside KENDAscope may possibly be set up.

Other

A proposal for a 4-year project starting in 2021 with realistic chances to be funded has been submitted by DWD on the exploration of artificial intelligence and neural networks for the estimation and specification of observation errors.

6. Description of individual tasks

Task 1. Algorithmic developments

Core developments related to the atmospheric data assimilation scheme, using the standard conventional or the reference (operational) set of observations. Besides the main analysis algorithm itself, these developments also relate to all kinds of components necessary for operational data assimilation, including quality control, bias correction, specification of observation errors, etc. (unless they are developed for a particular observing system).

Task 1.1: Refinements/extensions of reference KENDA

Refinement and further development of the reference KENDA system wherever it is found to be required and appropriate depending on results. The reference KENDA is currently based on LETKF, but could include EnVar or PF in the later stages of the project.

General issues continue to be the ensemble size, analysis update frequency, covariance inflation, localisation, specification of observation errors, etc.

Candidates for likely implementation and assessment are:

Some further ideas include:

The task will likely also include the implementation and assessment of machine learning techniques for different parts of the processing (potentially e.g. for quality control, observation error specification, bias correction, observation operator components).

(Work plan at COMET for 2021:improve use of observations in high-resolution KENDA analysis, looking at statistics of observations and analysis increments; test use of radar volumes and GPS ZTD/STD.)

Deliverable:

Optimised KENDA system.

Date of delivery:

Aug. 2025, since this is an ongoing task.

Task 1.2: Variational DA (EnVar)

Development of variational data assimilation (Var) for the convective scale, namely:

Note that the variational scheme provides only a single analysis for a deterministic forecast (unless a whole ensemble of variational analyses is run). However, it is possible to apply ‘centering’ of the analysis ensemble around the variational deterministic analysis (this replaces the analysis ensemble mean by the deterministic analysis). Previous tests with this kind of centering for the global system where the deterministic analysis has a higher resolution than the ensemble have been promising. The benefit has yet to be shown for the convective scale and with deterministic and ensemble runs at the same resolution.

Major work packages for 3DVar/3D-EnVar include:

Deliverables/Date of delivery:

 

Task 1.3: Particle filter (PF)

To address non-Gaussianity, research is continued on the refinement and optimization of the PF to complement or replace the LETKF core for the computation of the transform matrix in the ensemble filter in DACE. This work will not only be done with ICON-LAM but to a rather large extent with toy models and with the global ICON model.

Deliverable/Date of delivery:

Task 2. Observations (from surface to clouds)

Implementation and optimization of the use of observation types other than the original set of conventional observations. Of special interest is high-resolution information in the planetary boundary layer (PBL, e.g. on low-level convergence), but also on cloud and precipitation. In general, the deliverables are forecast improvements as a result of the use of these observations.

In the advent of corresponding projects, additional sub-tasks on further observation types may be added, e.g. on AMSU (ATMS, IASI,…) radiances from polar orbiting satellites (interest of ARPAE after 2022).

Task 2.1: Radar data

Refinement of the use of these data. Some (potential) issues, mostly related to radar reflectivity Z, are:

Deliverable:

Operationalisation and improved forecast impact.

Available FTE per year:

0.4 by Kobra Khosravian (until Q1/2023, DWD), 0.2 by Klaus Stephan (DWD), 0.1 by Uli Blahak (DWD); 0.6 by Virginia Poli (ARPAE), 0.8 by Thomas Gastaldo (until he leaves for a post-doc in Canada; ARPAE).
There is a risk that the FTE’s are cut down if Thomas Gastaldo leaves for a post-doc position in Canada before the end of his contract at ARPAE.

Task 2.2: Ground-based GNSS data (STD + ZTD:slant + zenith total delay)

Operationalisation and refinement of the use of these data. (Potential) issues are:

Deliverable:

Capability of using ground-based GPS data beneficially and operationally in the LETKF and in VAR.

Task 2.3: SEVIRI + FCI (MTG IMAGER) all-sky (cloudy) IR + VIS radiances

Direct use of all-sky (cloudy) SEVIRI mainly in view of assimilating cloud information (horizontal distribution, cloud top height). Scheduled to start operations in autumn 2023, the first MTG imager FCI will thereon complement the MSG SEVIRI data.

Data from 3 types of channels are available (and at least the first 2 will be addressed):

  1. Reflectance from visible channels (VIS), providing information during daytime mainly on cloud cover, in particular of low clouds not seen well by other channels; this sub-task will build on previous work done at DWD and HErZ at LMU Munich (e.g. Scheck et al., 2020).
  2. Brightness temperature from infrared (IR) water vapour (WV) channels, providing information related to water vapour and/or cloud cover and cloud top height of high and mid-level clouds all day; this sub-task will extend the experimental use of clear-sky WV data developed in KENDA-O (Hutt et al., 2020) to an all-sky approach.
  3. Brightness temperature from infrared (IR) window channels, providing information related to cloud cover and cloud top height of all cloud types all day.

Particular issues:bias correction, observation error specification, non-Gaussian distribution of first guess departures, non-linearity, localisation (for non-local observations), thinning, tailored covariance inflation, etc.; further potential approaches may include variable transformation (non-Gaussianity), smoothing and warping (non-linearity, displacement errors), etc; an important aspect is also the combination of the different channels.

A further issue is the adaptation to VAR, and this includes the need of the TL and adjoint of the forward operator. (Note that in VAR, the forward operator has to be formulated such that the model equivalent to any observation is computed solely from those prognostic model fields (called control vector) which are updated in the analysis. If the forward operator requires additional prognostic or diagnostic fields as input then these fields have to be treated as fixed values or as sink variables. Sink variables may vary during the variational procedure, but the information on the modified values of the sink variables is not conveyed to the analysis state.) For the satellite radiances, the forward operator consists of two parts:Firstly, the radiative transfer (RT) model (MFASIS/RTTOV) for which the TL and adjoint have already been developed. Secondly, the computation of the input fields of the RT model (e.g. cloud fraction) from the prognostic model variables. This computation should reflect the diagnostics e.g. of cloud cover as described in the NWP model and in particular its radiation parameterization as well as possible. In collaboration with WG3, this will be done by using the formulations of these parameterizations (or approximations thereof), or alternatively by using neural networks. Another pivotal question for the adaptation to VAR is whether to include hydrometeor mixing ratios in the control vector (for the minimization) or not.

Deliverable:

Capability of using all-sky VIS + IR data with sustained benefit and operationally in KENDA.

Task 2.4: MTG IRS (Meteosat Third Generation Infrared Sounder)

Even though the ultimate goal might be the use of all-sky data, the main benefit from the use of these multi-spectral radiances on top of the cloud information from the imagers (Task 2.3) is expected from profile information on temperature, humidity, and stability. Therefore, the focus will lie on the use of clear-sky radiances. Later, this can be extended to radiances sensitive to regions above cloud or affected (slightly) by clouds, always with the main aim to derive information on temperature and humidity.

The IRS instrument on board of MTG-S1 is currently scheduled to start operations no sooner than autumn 2024 (with first test data likely available half a year before that). In order to prepare the use of these data, test data for IRS can be used for development:

Research topics/issues include:

Deliverable:

Assimilation in a first usable version.

Task 2.5: Screen-level observations

Use of local observations from surface stations. Primarily, this means use of 2-m temperature, 2-m humidity, and 10-m horizontal wind observations from Synop stations, which are considered potentially beneficial for describing the (pre-) convective environment (low-level convergence) and very low stratus.

In addition, this task may also include observations from other types of stations, e.g. roadside sensors (with temperature/humidity/wind at e.g. 2, 4, 5, 7, 10 m above the ground).

Particular issues: Station-dependent conditional nonlinear online bias correction (see Otkin and Potthast, 2019), specification of observation error and quality control, vertical localisation, observation operators for observations on levels other than 2 m resp. 10 m above the ground; TL + adjoint of observation operators for VAR.

Deliverable:

Capability of using screen-level observations beneficially in KENDA.

Task 2.6: Boundary-layer profiling observations

Development of assimilation of a variety of data types which typically deliver profile information (i.e. data at several vertical levels) mainly in the planetary boundary layer. These include:

  1. ground-based microwave radiometer (MWR) radiances, providing information on (a.o.) temperature and humidity with low vertical resolution; the focus is on the direct assimilation of the radiances rather than retrievals (temperature, humidity, IWV, liquid water).

    Deliverable: Implementation and assessment of assimilation of MWR direct radiances in the LETKF.

  2. Doppler wind lidar (wind profiles)

    Deliverable: operational applicability.

  3. Raman lidar (temperature + humidity profile)

    Deliverable: Development and test of observation operator (using mixing ratio as humidity variable), assimilation experiments

  4. drones

    Deliverable: Assessing impact of assimilation of drone observations.

  5. towers

    Deliverable: Technical implementation, monitoring, assimilation tests depending on monitoring results

Other observation types may be added when available, e.g. DIAL lidars (humidity profiles and potentially 3-D scans; some FTE e.g. for observation monitoring could be taken from Jasmin Vural (DFG/DWD) for this).

Task 3. Soil/surface

The main topics here are soil moisture analysis (SMA) or initialization (Tasks 3.1, 3.2), sea surface temperature (SST) analysis (including analysis of sea ice) (Task 3.3), and snow analysis (namely analysis of snow depth, cover and/or snow water equivalent).

However, the snow analysis is not part of this project. Instead, it is considered more suitable to be addressed in a Priority Project on its own because it is more closely related to (other tasks in) WG3b. Having a separate PP for this topic will facilitate the management of the work and communication between the staff involved.

Further sub-tasks may be added during the course of the PP if resources can be allocated for them. A candidate might be a soil temperature analysis, possibly including the use of satellite-derived land surface temperature data.

Task 3.1: Soil moisture analysis using satellite soil moisture data

Use of satellite-derived soil moisture products, generated by HSAF from MetOp scatterometer (ASCAT) data and possibly from L-band microwave imagers (SMOS, SMAP), in the LETKF. Assimilating such data with the same scheme (e.g. LETKF) as for the atmospheric variables enables the system for coupled DA:atmospheric observations (e.g. of 2-m temperature) may be used to influence also the soil, while the soil moisture data may influence the atmospheric analysis as well. Technically, one or several additional analysis levels for the soil have to be added to the control vector of the LETKF.

Issues: situation-dependent vertical localization between soil and atmosphere depending on realism of ensemble background error correlations across surface, observation error specification, bias correction, etc.

The work plan for 2021 consists of:

Deliverable:

see text.

Task 3.2: Soil moisture analysis/initialization

Currently, a stand-alone variational ‘soil moisture analysis’ (SMA) scheme using daytime 2-m temperature observations (interpolated to the model grid) as input is applied to the global ICON / ICON-EU nest. The soil moisture of the convective-scale configuration at DWD, ICON-D2, is then nudged towards the (interpolated) ICON-EU soil moisture, which has the same soil model (TERRA).

In this task, it is planned to re-write the SMA in the DACE framework. There are several possibilities to derive the sensitivity of daytime 2-m temperature to soil moisture (in the morning):a) using the ensemble covariances (LETKF approach), b) by means of additional forecasts with perturbed initial soil moisture (approach formerly adopted for COSMO-EU), or c) by parameterization (based on latent heat fluxes, approach used for the global ICON/ICON-EU). It has yet to be decided which approach(es) should be implemented (and which one first).

The primary goal of the task is to implement a first approach for testing and to compare it to the current nudging towards the interpolated ICON-EU soil moisture. Note that this implies the need to re-write also the 2-m temperature gridded analysis based on Synop observations in the DACE framework; a variational approach is envisaged.

Later on, the task might be extended in one or several ways:further refinement of the implemented scheme, implementation of another approach, use of further observations (e.g. satellite data related to soil moisture), etc.

Note that as long as only atmospheric observations are used in this scheme, one may prefer to speak of a soil moisture initialization rather than analysis.

Deliverable:

Implementation and test of a first version for new soil moisture analysis/initialization in DACE.

Task 3.3: Sea surface temperature (SST) analysis

Currently, a stand-alone SST analysis is run for COSMO-D2/ICON-D2. In-situ data are used in a simple Cressman-type successive correction method to correct the background field which is based on the Ostia analysis from the Met Office.

In this task, it is planned to completely re-write the SST analysis in the DACE framework and replace the old scheme by a 2-d variational scheme. (This implies to implement a 2-dimensional version of the (currently 3-dimensional) B-matrix.)

Deliverable:

Implementation and operationalization of a new SST analysis in DACE.

7. References

Benjamin SG, Weygandt SS, Brown JM, Hu M, Alexander CR, Smirnova TG, Manikin GS. 2016.
A North American hourly assimilation and model forecast cycle:the Rapid Refresh.
Mon. Wea. Rev., 144:1669-1694, doi: 10.1175/MWR-D-15-0242.1.

Bick T, Simmer C, Trömel S, Wapler K, Hendricks Franssen H, Stephan K, Blahak U, Schraff C, Reich H, Zeng Y, Potthast R. 2016.
Assimilation of 3D radar reflectivities with an ensemble Kalman filter on the convective scale.
Q. J. R. Meteorol. Soc. 142:1490-1504, doi:10.1002/qj.2751

Buehner M. 2005.
Ensemble-derived stationary and flow-dependent background-error covariances:Evaluation in a quasi-operational NWP setting.
Q. J. R. Meteorol. Soc., 131:1013-1043, doi:10.1256/qj.04.15

Gastaldo T, Poli V, Marsigli C, Alberoni PP, Paccagnella T. 2018.
Data assimilation of radar reflectivity volumes in a LETKF scheme.
Nonlin. Processes Geophys., 25:747-764, doi:10.5194/npg-25-747-2018.

Geer AJ, Lonitz K, Weston P, Kazumori M, Okamoto K, Zhu Y, Liu EH, Collard A, Bell W, Migliorini S, Chambon P, Fourrié N, Kim M-J, Köpken-Watts C, Schraff C. 2018.
All-sky satellite data assimilation at operational weather forecasting centres.
Q. J. R. Meteorol. Soc. 144: 1191-1217, doi:10.1002/qj.3202.

Gustafsson N, Janjic T, Schraff C, Leuenberger D, Weissmann M, Reich H, Brousseau P, Montmerle T, Wattrelot E, Bučánek A, Mile M, Hamdi R, Lindskog M, Barkmeijer J, Dahlbom M, Macpherson B, Ballard S, Inverarity G, Carley J, Alexander C, Dowell D, Liu S, Ikuta Y, Fujita T. 2018.
Survey of data assimilation methods for convective-scale numerical weather prediction at operational centres.
Q. J. R. Meteorol. Soc. 144:1218-1256, doi:10.1002/qj.3179.

Harnisch F, Weissmann M, Periáñez Á. 2016.
Error model for the assimilation of cloud-affected infrared satellite observations in an ensemble data assimilation system.
Q. J. R. Meteorol. Soc., 142: 1797-1808, doi:10.1002/qj.2776.

Hunt BR, Kostelich EJ, and Szunyogh I. 2007.
Efficient data assimilation for spatiotemporal chaos:a Local Ensemble Transform Kalman Filter.
Physica D, 230:112-126, doi:10.1016/j.physd.2006.11.008.

Hutt A, Schraff C, Anlauf H, Bach L, Baldauf M, Bauernschubert E, Cress A, Faulwetter R, Fundel F, Köpken-Watts C, Reich H, Schomburg A, Schroettle J, Stephan K, Stiller O, Weissmann M, Potthast R. 2020.
Assimilation of SEVIRI water vapour channels with an ensemble Kalman filter on the convective scale.
Frontiers in Earth Science, 8, 70, doi:10.3389/feart.2020.00070.

Kurzrock F, Cros S, Ming F, Otkin J, Hutt A, Linguet L, Lajoie G, Potthast R..2018.
A review of the use of geostationary satellite observations in regional-scale models for short-term cloud forecasting.
Meteorol. Zeitschrift 27: 277-298, doi:10.1127/metz/2018/0904.

Lei L, Whitaker JS, Bishop C. 2018.
Improving assimilation of radiance observations by implementing model space localization in an ensemble Kalman filter.
J. Adv. Model. Earth Syst. 10:3221-3232, doi: 10.1029/2018MS001468

Leuenberger D, Haefele A, Omanovic N, Fengler M, Martucci G, Calpini B, Fuhrer O, Rossa A, 2020.
Improving high-impact numerical weather prediction with lidar and drone observations
BAMS, in press doi:10.1175/BAMS-D-19-0119.1.

Milan M, Macpherson B, Tubbs R, Dow G, Inverarity G, Mittermaier M, Halloran G, Kelly G, Li D, Maycock A, Payne T, Piccolo C, Stewart L, Wlasak M. 2020.
Hourly 4D-Var in the Met Office UKV operational forecast model.
Q. J. R. Meteorol. Soc. 146:1281-1301, doi:10.1002/qj.3737.

Necker T, Weissmann M, Ruckstuhl Y, Anderson J, Miyoshi T. 2020.
Sampling error correction evaluated using a convective-scale 1000-member ensemble.
Mon. Wea. Rev., 148:1229-1249, doi:10.1175/MWR-D-19-0154.1.

Otkin J, Potthast R. 2019.
Assimilation of all-sky SEVIRI infrared brightness temperatures in a regional-scale ensemble data assimilation system.
Mon. Wea. Rev. 147:4481-4509, doi:10.1175/MWR-D-19-0133.1

Potthast R, Walter A, Rhodin A. 2019.
A Localized Adaptive Particle Filter (LAPF) within an operational NWP framework.
Mon. Wea. Rev. 147:345-362, doi:0.1175/MWR-D-18-0028.1.

Scheck L, Weissmann M, Bach L. 2020.
Assimilating visible satellite images for convective-scale numerical weather prediction: A case study.
Q. J. R. Meteorol. Soc. 146:accepted, doi:10.1002/qj.3840.

Schomburg A, Schraff C, Potthast R. 2015.
A concept for the assimilation of satellite cloud information in an Ensemble Kalman Filter:single-observation experiments.
Q. J. R. Meteorol. Soc. 141:893-908, doi:10.1002/qj.2407.

Schraff C, Reich H, Rhodin A, Schomburg A, Stephan K, Periáñez A, Potthast R. 2016.
Kilometre-scale ensemble data assimilation for the COSMO Model (KENDA).
Q. J. R. Meteorol. Soc. 142:1453-1472, doi: 10.1002/qj.2748.

Stephan K, Klink S, Schraff C. 2008.
Assimilation of radar-derived rain rates into the convective-scale model COSMO-DE at DWD.
Q. J. R. Meteorol. Soc. 134:1315-1326, doi:10.1002/qj.269.

Wattrelot E, Caumont O, Mahfouf JF. 2014.
Operational implementation of the 1D+3D-Var assimilation method of radar reflectivity data in the AROME model.
Mon. Wea. Rev. 142:1852-1873, doi:10.1175/MWR-D-13-00230.1.

Wu W-S, Parrish DF, Rogers E, Lin Y. 2017.
Regional ensemble-variational data assimilation using global ensemble forecasts.
Wea. Forecasting, 32:83-96, doi:10.1175/WAF-D-16-0045.1.

8. Appendix: Task table

Planned resources in the COSMO-year 2021:

Germany:

DWD: Hendrik Reich 0.8 FTE, Christoph Schraff 0.4 FTE, Klaus Stephan 0.4 FTE, Roland Potthast 0.2 FTE, Harald Anlauf 0.1 FTE, Elisabeth Bauernschubert 0.4 FTE, Stefanie Holborn 0.4 FTE, Mareike Burba (IVS) 0.8 FTE, Nora Schenk 0.5 FTE, Gernot Geppert 0.3 FTE, Uli Blahak 0.1 FTE, Michael Bender 0.2 FTE, Lilo Bach (MetBw) 0.6 FTE, Annika Schomburg 0.3 FTE, Thomas Deppisch (IAFE-Sinfony) 0.4 FTE, Christine Sgoff (Gridcast) 0.3 FTE, Kobra Khosravian (IVS) 0.4 FTE, Jasmin Vural (DFG) 0.8 FTE, N.N. (IAFE-MTG) 0.5 FTE
in total 7.9 FTE

Switzerland:

MeteoSwiss: Daniel Leuenberger 0.5 FTE, Claire Merker 0.7 FTE, Samuel Monhart 0.3 FTE, N.N. (Project EMER-Met on wind lidar) 0.1 FTE, N.N. (Project OWARNA on Raman Lidar) 0.7 FTE
in total 2.3 FTE.

Italy:

COMET: Francesca Marcucci 0.1 FTE, Lucio Torrisi 0.1 FTE, Valerio Cardinali 0.2 FTE
ARPAE: Virginia Poli 0.6 FTE, Thomas Gastaldo 0.8 FTE
in total 1.8 FTE.

Remarks

Resources in 2021 meet the requirements.

Task Contributing scientist(s) FTE
total
FTE
2021
Start Deliverables Date of
delivery
Preceding
tasks
1.1 Reich, Stephan, Potthast, Anlauf, Schraff (DWD)
Leuenberger (MCH)
Torrisi, Marcucci, Cardinali (COMET)
7.5 1.45
1.1
0.3
09/20 Refinements/extensions of reference KENDA 08/25 -
1.2 Bauernschubert, Holborn, Burba, Schraff, Potthast (DWD) 7 1.8 09/20 Variational DA (EnVar):(C)EnVar
3DVar
4D-EnVar
06/21
12/21
08/25
-
1.3 Schenk (DWD) 4 0.5 09/20 Particle Filter 08/25 -
2.1 Khosravian, Stephan, Blahak (DWD)
Poli, Gastaldo (ARPAE)
7.2 0.7
1.4
09/20 Radar reflectivity Z + radial velocity Vr
Vr at ARPAE
08/25
12/21
-
2.2 Bender (DWD) 1 0.2 11/20 GNSS ZTD + STD 12/22 c -
2.3 Bach, Schomburg, Deppisch (DWD) 1.5 1.3 09/20 SEVIRI/FCI cloudy radiances VIS
IR WV
12/21 c
12/22 c
-
2.4 N.N. (DWD/IAFE) 4 0.5 01/21 MTG IRS 08/25 -
2.5 Sgoff (DWD) 1.5 0.3 09/20 Screen-level obs RH2m
uv10m, T2m, etc
03/21 c
08/25
-
2.6 Leuenberger, Merker, Monhart, N.N1.
N.N2 (MCH)
Vural, Schraff (DWD)
6.5 2.2
0.85
09/20 PBL profiling obs MWR
wind lidar
Raman lidar
06/23
09/22
09/22
-
3.1 Cardinali (COMET) 1 0.1 09/20 Soil moisture analysis using satellite soil moisture data real-time tests 08/25
08/21
-
3.2 Geppert, Lange (DWD) 0.6 0 01/22 Soil moisture analysis/initialization 12/22 -
3.3 Geppert (DWD) 0.2 0.2 01/21 SST analysis 12/21 -