1 / 25

ESMValTool for Benchmarking Models with ESA CCI Data

ESMValTool for Benchmarking Models with ESA CCI Data. Mattia Righi 1 , Veronika Eyring 1 , Axel Lauer 1 , Alexander Löw 2 , Benjamin Müller 2 , Daniel Senftleben 1 , Sabrina Wenzel 1 , Martin Evaldsson 3 , Ulrika Willén 3 , and Yoko Tsushima 4 CMUG WP 5.1 Contribution

aobrien
Download Presentation

ESMValTool for Benchmarking Models with ESA CCI Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ESMValTool for Benchmarking Models with ESA CCI Data Mattia Righi1, Veronika Eyring1, Axel Lauer1, Alexander Löw2, Benjamin Müller2, Daniel Senftleben1, Sabrina Wenzel1, Martin Evaldsson3, Ulrika Willén3, and Yoko Tsushima4 CMUG WP 5.1 Contribution 1Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Oberpfaffenhofen, Germany 2Department of Geography, University of Munich (LMU), Germany 3 Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden 4 Met Office, Exeter, United Kingdom CCI CMUG Integration Meeting Munich, 14-16 March 2016

  2. Motivation and Goal • Model benchmarking initiatives have become increasingly important to evaluate the quality of coupled Earth System Models (ESMs) and to support the model development process. • However, ESA data is currently not used in the context of routine model evaluation. • Model benchmarking is also important for the Climate Research Groups (CRGs) within the CCI and various CMUG activities. • CMUG will therefore establish a standardized model benchmarking approach for the CCI, and will contribute to the development of a community-wide Earth System Model Evaluation Tool (ESMValTool) that is currently developed by different partners in different projects. • The goal is to work towards a standardized community based benchmarking toolkit that includes ESA CCI data and that could be used operationally for CMIP6 analysis in the forthcoming years. It is expected that the preparation of the individual ESA CCI datasets in CMIP compliant format for Obs4MIPs as well as the corresponding technical note for Obs4MIPs are created and submitted by the individual ESA CCI teams.

  3. Overview ESA CMUG WP 5.1 Tasks The aim of this WP is to contribute to the development of a standardized community based evaluation and benchmarking toolthat makes full use of the novel ESA CCI data products and that could be used for CMIP model analysis in the forthcoming years by the climate research community. Within the ESA CCI programme, the ESMValToolwill be provided to the Climate Research Groups for their work within the ESA-CCI programme. This task will also provide comparison to other observations that are routinely used in model evaluation, in particular those that are contributed to obs4mips. D5.1 v1: Initial version of the ESMValTool with one ESA CCI dataset for test purposes shared among CMUG partners [DLR, LMU, June 2015]. D5.1 v2: Advanced version of the ESMValTool with ESA CCI datasets and user guide released to CMUG and ESA CCI teams [DLR, LMU, June 2016]. D5.1 v3: Final version of the ESMValTool with 10 ESA CCI datasets and user guide released to wider community [DLR, LMU, June 2017].

  4. ESMValTool Version 1.0 released with ESA CMUG contributions http://www.esmvaltool.org/ Eyring et al., Geosci. Model Dev. Discuss, 8, 7541-7661, 2015

  5. Current Status: Contributing Institutions(currently ~60 developers from 22 institutions and ~40 users) • Deutsches Zentrum für Luft- und Raumfahrt (DLR), Institut für Physik der Atmosphäre, Germany • Swedish Meteorological and Hydrological Institute (SMHI), Norrköping, Sweden • Agenzia nazionale per le nuove tecnologie, l’energia e lo sviluppo economico sostenibile (ENEA), Italy • British Atmospheric Data Centre (BADC) , UK • Centre for Australian Weather and Climate Research (CAWCR), Bureau of Meteorology, Australia • Deutsches Klimarechenzentrum (DKRZ), Germany • ETH Zurich, Switzerland • Finnish Meteorological Institute, Finland • Geophysical Fluid Dynamics Laboratory (GFDL) NOAA, USA • Institut Pierre Simon Laplace, France • Ludwig Maximilian University of Munich, Germany • Max-Planck-Institute for Meteorology, Germany        • Met Office Hadley Centre, UK • Météo France, Toulouse, France • Nansen Environmental and Remote Sensing Center, Norway • National Center for Atmospheric Research (NCAR), USA • New Mexico Tech, USA • Royal Netherlands Meteorological Institute (KNMI), The Netherlands • University of East Anglia (UEA), UK • University of Exeter, Exeter, UK • University of Reading, UK • University of Wagingen, The Netherlands

  6. Maintenance, Documentation and Technical Development 1. Maintenance • A subversion controlled repository has been made available to allow the development by multiple users. In addition, a Mantis bug tracking system and a wiki page is provided to facilitate communication and documentation of the tool. • First release of ESMValTool version 1.0 as open source software in December 2015 • The ESMValTool coredevelopmentteam (DLR, LMU, and SMHI) isresponsibleformaintaining a stableversionofthetool in thetrunkincludingqualitycontrolandtesting. 2. Documentation • ESMValTool documentation paper published in GMDD in August 2015, revisions pending • A first version of the user guide has been published in GMDD and will be further revised for the final version to support the ESA CCI teams in the use of the tool • In addition, ESMValTool developers’ internal wiki page that documents new developments and progress 3. Technical development • First ESMValTool technicalworkshophasbeen hold (March 2015, Munich) forthedefinitionofroadmaps; severaladditionlmeetingsbetween DLR and LMU • Automatedtestingofdiagnostics (work in progress) • Visualize the results through a website (work in progress) • ESMValTool performanceimprovements (work in progress)

  7. Development and integration of ESA CCI data into the ESMValTool

  8. Performance Metrics calculated with ESA CCI Data [DLR] NEW: ESA CCI total column ozone ESA CCI AOD 550 nm already implemented in ESMValTool version 1.0 Implementation of other ESA CCI datasets planned • Clouds & Radiation • Aerosol+ • XCO2 • SIC and SIT • Ocean colour • SSTs • Land Cover • Fire • Soil Moisture Eyring et al., ESMValTool (v1.0), GMDD, 2015 • Relative error measures of CMIP5 model performance, based on the global seasonal-cycle climatology (1980–2005) computed from the historical CMIP5 experiments. Similar to Figure 9.8 of IPCC AR5 (Flato et al., 2013 and Gleckler et al. (2008)). • A similar figure will be produced for selected ESA CCI ECVs using ESA CCI as the reference dataset and if available an alternate observational data set for comparison.

  9. Goal: Run ESMValTool alongside the ESGF for Routine Evaluation of CMIP6 Models Goal: ESMValTool as one of the CMIP documentation functions to routinely assess the performance of CMIP DECK and the CMIP6 Historical Simulations, running alongside the ESGF More routine usage of ESA CCI data for ESM evaluation studies through a community based evaluation and benchmarking tool Eyring et al., CMIP6 Overview, GMDD, 2015

  10. ATMOSPHERE: Clouds [SMHI] (1) • Preliminary results using Cloud-CCI and CLARA-A2 cloud cover for the “Performance metrics”. • Annual climatology for observations, reanalysis and historical CMIP5 experiments (1982-2005). • BIAS figs: models compared to CCI-cloud. Relative errors: Cloud CCI upper/CLARA lower triangle CCI Cloud Cover has stronger minima and maxima and larger values at high latitudes over sea compare to the models and CLARA-A2 • clara • cci • Mean Median Can ERAINT GFDL IPSL MPI LR MPI MR NCEP2 • model model ESM2 ESM2G CM5lr ESMlr ESMmr • ERAINT-CCI • NCEP-CCI • CanESM-CCI • CanESM-CCI • IPSL-CCI • GFDL-CCI • MPImr-CCI

  11. ATMOSPHERE: Clouds [SMHI] (2) Preliminary results using ESMvaltool Taylor diagram diagnostics showing the multi-year annual average performance of individual CMIP5 models and the multi-model mean in reproducing ESA-CCI Cloud cover (AVHRRv1.4), for common model and observational period 1982-2005 ERA-Interim CLARA-A2 Reference point Cloud_CCI CCI cloud cover has larger variability than the models...

  12. ATMOSPHERE: Teleconnections Clouds & SST [SMHI] Preliminary results using ESMvaltool Teleconnection diagnostics for the correlation between Nino3.4 CCI-SST and CCI -Cloud cover, and SST/ Clouds in two AMIP simulations,1982-2008 ANN-mean Cloud-Nino3.4 SST teleconnections CCI SST&Clouds Observed and modelled teleconnection pattern similar, but the models show hint of “double” ITCZ EC-Earth - CCI correlations EC-Earth SST&Clouds IPSL - CCI correlations IPSL SST&Clouds Difference in models and observed correlations

  13. ATMOSPHERE: Ozone[DLR] Comparison of total column ozone in CMIP5 models with ESACCI and NIWA • Seasonal climatology and time series for the period 1997-2005 • Overall good agreement between ESACCI and NIWA data

  14. ATMOSPHERE: Aerosols [DLR] (1) Overall good agreement between ESACCI and AERONET Overestimate of AOD in some stations (Sahara, Arabia, South America). Good agreement in marine stations.

  15. ATMOSPHERE: Aerosols [DLR] (2) OCEAN Evaluation of aerosol optical depth (AOD) at 550 nm in the CMIP5 models Model spread is large (~0.05 - 0.20). Differerence between MODIS Collection 6 data and ESA CCI product also significant. GLOBAL

  16. ATMOSPHERE: GreenhouseGases [DLR] (CO2) Preliminary comparison of column averaged atmospheric CO2 (XCO2) in CMIP5 models with ESA CCI data: • Data available for 2003-2013 • CMIP5 models tend to overestimate XCO2 from ESACCI XCO2 • Averaging kernels not yet used; more detailed comparison to follow. region 60N – 90N

  17. OCEAN: SeaIce [DLR] • Evolution of Arctic (top) and Antarctic (bottom) summer sea ice extent, representing the accumulated sea ice area of all grid cells with at least 15% sea-ice coverage • Generally large spread between different sea-ice observational datasets • ESACCI data in terms of sea ice extent rather on the high end • ESACCI-SSMI and -AMSR relatively similar

  18. TERRESTRIAL: SoilMoisture [LMU] (1) Comparisonofsoilmoisturespatialpatternsand PDFs betweenmodelsand ESA-CCI soilmostureproduct (Loew et al., 2013) Percentile spatial correlation MODEL BIAS

  19. TERRESTRIAL: SoilMoisture [LMU] (2) Covariability between soil moisture and precipitation dynamics in models and ESA CCI soil moisture product Anomaly correlation between precipitation and soil moisture for satellite CDRs and reanalysis data (Loew et al., 2013)

  20. TERRESTRIAL: Land Cover [LMU] (1) Challenge: no consistent PFT‘s across different CMIP models comparison of broad classes (tree, shrubs, bare, water); preprocessing using LC CCI user tools( Brovkin et al., 2013)

  21. TERRESTRIAL: Land Cover [LMU] (2) Snow has major impact on surface albedo and surface net radiation budget. Major discrepancies in climate models existing (snow albedo, seasosnality) (e.g. Hagemann et al., 2013; Loew et al., 2016) ESA CCI land cover conditions is used to evaluated snow dynamics Surface net radiation difference (model – observations)

  22. ATMOSPHERE: Clouds Auto-Assess [MetOffice] (1) • Aims: To produce a set of metrics & diagnostics to inform model development • Current status: • Aim to produce a first (alpha) external release to Unified Model partners in April 2016 • Currently assumes native Unified Model file format (pp) • A beta release around the end of the year. In the beta release we hope to be using CF standard names where they exist • Auto-assess Clouds & Radiation to be linked to ESMValTool • Data Format • Function to read standard names have been added • Confirmed to read in CERES-EBAF, ISCCP D2 in netcdf. Aim to read in other data in Obs4MIPs • What should we do with diagnostics with no CF standard names? • Interface • Integration with shallow link to be done for D5.1 v3 June 2017

  23. ATMOSPHERE: Clouds Auto-Assess [MetOffice] (2) - Auto-assess Clouds and Radiation summary plot -

  24. Summary and Outlook • ESMValTool documentation (user guide) published • Further important technical improvements identified and work underway (e.g. performance improvements) • Work on the implementation of ESA CCI ECVs so far mostly focused on studying these individually • Next step: inclusion of additional ECVs in performance metrics plot and more in depth analysis • D5.1 v2: Advanced version of the ESMValTool with ESA CCI datasets and user guide released to CMUG and ESA CCI teams [DLR, LMU, June 2016]. • Planned contribution from WP 5.1 to Remote Sensing of Environment Special Issue on Earth Observation of Essential Climate Variables • Performance Metrics Plot with CMIP5 models using ESA CCI datasets as reference dataset and alternate observations where possible • Individual sections for each implemented ESA CCI with details on the analysis and discussion of possible issues for climate model evaluation

More Related