Enhancements for Hydrological Modeling in ESMF

1. Enhancements for Hydrological Modeling in ESMF Cecelia DeLuca/NCAR (ESMF) December 19, 2008 AGU Fall Meeting

2. Outline • ESMF overview • Results and projects relevant to the hydrological community • ESMF + data services and semantic technologies • A peek at the future

3. Earth System Modeling Framework (ESMF) Status Initiated by NASA in 2002 and developed and managed by a multi-agency consortium More than 76 ESMF components in the community, used by the NASA Modeling, Analysis and Prediction Program, the Department of Defense Battlespace Environments Institute, the National Weather Service, and many others • Component-based framework that targets tightly coupled, computationally demanding weather and climate applications written in Fortran • ESMF provides standard interfaces for model components • ESMF provides common utilities and tools for routine modeling functions such as regridding between components

4. ESMF Application Example Part of the GEOS-5 Atmospheric General Circulation Modeling system • Each box is an ESMF gridded component or coupler component • ESMF State objects carry data between components • Every component (including couplers) has a standard interface to facilitate exchanges • The ESMF architecture enables the assembly of many different systems

5. Interoperability and standardization of interfaces There are only three ESMF component methods: Initialize, Run, and Finalize (I/R/F) Users associate their native code I/R/F methods with an ESMF Component through a registration process I/R/F methods cascade down the tree Small set of standard arguments: call ESMF_CompRun (myComp, importState, exportState,clock, phase, blockingFlag, rc)

6. Outline • ESMF overview • Results and projects relevant to the hydrological community • ESMF + data services and semantic technologies • A peek at the future

7. Data representation options 1. Representation in index space (Arrays) • One or more logically rectangular tilesstore indices and topology • Sparse matrix multiply for regridding withuser supplied interpolation weights • Very general data representation, butlimited interoperability since not muchsemantic info is encoded by the framework 2. Representation in physical space (Fields) • Built on Arrays + some type of Grid • Regridding using parallel on-line interpolation weight generation, bilinear or higher order • Standard high-level data structures increase interoperability Supported Array distributions

8. Coupling options in ESMF Multiple executable options • Lots of flexibility incoupling approaches • Single executable • Multiple executable • Couplingcommunicationscan be set up and calledin a coupler • Coupling communications can be setup in a coupler but calleddirectly from within components – useful when it is inconvenient to return from a component in order to perform a coupling operation • Recursive components for nesting higher resolution regions • Ensemble management, including sequential execution of ensemble members Contributed by U Maryland Coupler Comp B Comp A Comp A Comp B Array send/recv

9. New Features and Applications • ESMF parallel regrid: • Grid-Grid, Grid-Mesh, Mesh-Mesh • Bilinear and higher order • ESMF v3.1.1 beta release (September 2008) includes • unstructured mesh class • observational data streams • on-line parallel regridding • standard metadata packets • Consortium of U.S. operational and research centers (NOAA, NASA, DoD) has entered pilot phase of development of multi-model global ensemble using ESMF components The Flow-Following Finite Volume Icosahedral Model (FIM) from NOAA GSD is converting to ESMF to couple to National Weather Service models NUOPC National Unified Operational Prediction Capability

10. Targeting greater interoperability • ESMF is a toolkit with many options for coupling, and ESMF compliance alone has not meant plug-and-play • Several groups have been creating additional guidelines, conventions, and constraints for increased interoperability • Example: a NUOPC operational consortium committee has reached detailed agreements on the following: • Componentization level - atm, ocean, land, ice, atm physics and dynamics • Clock - how time is passed between models and modules • Ownership – what models and modules are allowed to modify data held in common (e.g. aerosol species and concentrations) • Level of data representation – index space or physical space • Metadata – the description of the data, grid, clock, etc. • Coupling approach - NUOPC will provide a standard template. • Discussions are underway on IO, post-processing, and component invocation points

11. Parallel weight calculation and regrid:Results for ESMF in CCSM • dTAUx/dy - closely related to the curl of the wind stress, which drives the upper ocean circulation and is therefore important to represent accurately in climate simulations • Mapped from the T62 (2-degree) Community Atmospheric Model (CAM) grid to the gx1v5 (384x320 irregularly spaced) • Noise is calculated as deviation of a point from the sum of itself plus four neighbors • 33% reduction in noise globally compared to previous CCSM bilinear approach • CCSM is now using ESMF for their standard remapping Interpolation noise in the derivative of the zonal wind stress Interp. noise grid index in latitudinal direction Black = bilinear Red = higher-orderESMF v3.1.1 Green = higher order ESMF v4.0.0

12. Performance Portablity 30+ platform/compiler combinations regression tested nightly, new ports include gfortran (4.3), Solaris, and (almost) Windows ESMF Sparse Matrix Multiply Run-Time Performance at the petascale… The chart at right shows scaling of the ESMF sparse matrix multiply, used in regridding transformations, out to 16K processors. (ESMF v3.1.0rp2) Plot from Peggy Li, NASA/JPL Tested on ORNL XT4 in a variety of configurations . -N1 means 1 core per node.

13. Outline • ESMF overview • Results and projects relevant to the hydrological community • ESMF + data services and semantic technologies • A peek at the future

14. Metadata export from ESMF (v4.0) • ESMF holds information about grids, component inter-relationships • Users can associate additional metadata with ESMF classes (Grids, Arrays, Fields, States, Components) as name/value pairs in Attribute packages • Attribute packages can be standard (e.g CF conventions for Fields) or custom • Packages can be nested – for example, metadata for an import State contains metadata for all the Fields in the State. • Metadata can be exported in plain text or XML format … • then ingested into portal, where users can search and browse components and datasets via the portal, “trackback” to the model configuration that created a dataset, create dynamic comparison tables for specific properties, create private or shared workspaces, etc. • Excellent support for component and model inter-comparison projects. The path from model metadata to web interface involves many format translations along the way Image courtesy of Rocky Dunlap, Georgia Institute of technology

15. Numerical Techniques for Global Atmospheric Models Summer Colloquium Held at NCAR during summer 2008, about 80 participants 9 atmospheric dynamical core components archived Running 22 test cases at up to 5 resolutions Search, browse, trackback, comparison tables Automated upload of simulation data, publishing, and comparison Data search and browse via ESG Gateway Demonstrates how knowledge base can combines with ESMF http://dycore.ucar.edu Curator project sponsored by

16. Work in progress • Translation of ESMF interfaces into web service interfaces to support loose coupling, multi-language coupling, and integration of components into service oriented architectures (Curator project) • Integration of CCSM ESMF version with Kepler workflow tool (PhD student at Istanbul Technical University) • Automated coupler generation (PhD student at Georgia Tech) • New TeraGrid award: invoke CCSM ESMF version from an ESG-based science gateway on the TeraGrid - data and metadata from the run are automatically stored back to the portal to support intercomparison projects • Working with Cray, Microsoft and NASA to finish port to Windows HPC 2008 Image courtesy of Rocky Dunlap, Georgia Institute of technology

17. A peek at the future … 2009 2010 2011 2012 2013 2014 2015 Interoperability Compliance definitionsand verification tools Integration with workflows Generation of service interfaces for loose coupling Metadata links to data and other services Automated model assembly andrun environment Based on metadata files Tutorial development andclassroom deployment Numerical/computational maintenance and development