The Model Coupling Toolkit and the CCSM Next-Generation Coupler (NGC)

1. The Model Coupling Toolkit and the CCSM Next-Generation Coupler (NGC) J. Walter Larson and Robert L. Jacob Mathematics and Computer Science Division Argonne National Laboratory

2. Outline • Prologue • Introduction—The Accelerated Climate Prediction Initiative (ACPI) Avant Garde project • Description of the Model Coupling Toolkit (and status) • Overall Design of the new Community Climate System Model (CCSM) Next-Generation Coupler (NGC)

3. Prologue • High-performance is crucial for earth system modeling applications • A less obvious point—software quality is important, too: • Well-tested and documented • Modularity to reduce software complexity • Extensibility For simulation science, the software should embody the rigor and reproducibility of the scientific method

4. The CCSM • Community Climate System Model developed by the USA National Center for Atmospheric Research (NCAR) • A coupled earth system model comprising the following component models: • Atmosphere • Ocean • Sea Ice • Land-surface • Flux Coupler Long-term (2005) Goal: implement the CCSM in a high-performance software framework

5. From CCSM to a Framework… • Re-engineer component model code to maximize modularity • Render component models as components amenable to plug-and-play usage • Identify common utility functions that reside in multiple component models, and make them shared utilities • Absorb common flux coupler and shared utility routines part of the framework core This is a big job…

6. ACPI Avant Garde A Joint US Department of Energy / National Science Foundation project to apply modern software engineering practices to create a performance-portable version of the NCAR Community Climate System Model (CCSM) Participants: DOE Laboratories: ANL, LANL, LBNL, LLNL, and ORNL NCAR NASA Data Assimilation Office Web Site: http://www.mcs.anl.gov/acpi

7. ACPI Avant Garde Project Term: 1 June, 2000 – 31 September, 2001 Current Objectives: • Modular, high-performance atmosphere GCM with a choice of three dynamical cores • Create a new, modular, extensible and parallel flux coupler • Optimization of the ocean GCM POP • Implementation of parallel I/O in CCSM A grant application for DOE funding through 2006 to expand ACPI has been submitted

8. A Typical Coupled Model AGCM Temp, Rain, Radiation, Wind Sea Ice Temp, Albedo Land Surface Temp, Soil Moisture OGCM Temp, Currents, Salinity

9. Coupled Modeling System

10. Our Objectives • Create a model coupling environment that is: • Flexible • Extensible • Performance Portable—supports message-passing, shared-memory, and hybrid parallelism • Highly configurable and easy-to-use

11. The Flux Coupler The Coupler has two basic functions: • Command and Control • Data flow between component models (including grid interpolation, time averaging and accumulation, flux physics calculations, and merging of state and flux data from multiple component models). We need to support both functions, but need solutions that are highly extensible and configurable The model coupling toolkit (mct) is designed to support function number two above.

13. mpeu Provides the following services: • F90 module-style access to MPI • Portable/Flexible definition of types • Support for multiprocessor stdout/stderr • Error handling / shutdown • Timing/Load balance monitoring tools • Sorting Tools • Support for basic derived types on which low-level classes in the mct are built Code used as scaffolding, but utilities might fit well in CCSM.

14. The model coupling toolkit The model coupling toolkit provides a foundation for parallel coupled models, with general utilities for: • Data decomposition • Flexible, extensible, indexible field storage • Time averaging and accumulation • Field interpolation • Intercomponent communications and parallel data transfer The mct eases the construction of coupler compuational cores and component-coupler interfaces

15. The model coupling toolkit Software description: • Highly Modular, Implemented in Fortran 90 • Approximately 10,000 Lines of Code • Functions have extensive argument and error checking • Modules and routines have prologues that can be processeed using ProTeX to create LaTeX documentation • Model Coupling Toolkit Web Site: http://www.mcs.anl.gov/~larson/mct

16. MCT Assumptions (so far; subject to change) Each component has its own MPI communicator Each component has a unique integer ID (provided by MPH) Each component is on a distinct set of processors Interpolation implemented as sparse-matrix multiply Components can exchange only real and integer data as groups of vectors The mct user supplies: Consistent numbering schemes for grid points Interpolation Matrix Elements Any number of components, any grid, any decomposition, any number of processors-per component is allowed Features

17. 1. Data decomposition and indexing: Data decomposition descriptors: • GlobalMap (1-D) • GlobalSegMap(multidimensional) Local vector indexing—Navigator Global to Local and Local to Global indexing

18. 2. Flexible, extensible, indexible field storage—The AttrVect class • Internal field representation in the mct • Forms the basis for other classes including • Accumulator • GeneralGrid • SparseMatrix

19. 3. Time Averaging and Accumulation—TheAccumulatorclass: • Support for: • Time averaging • Time summation

20. 4. Grid Representation and Grid Interpolation: • Grid representation—GeneralGrid • Literal listing of grid point coordinates • Any dimensionality of grid supported • Gridpoint sorting supported • Extensible storage allows for inclusion of area and volume weights

21. 4. Grid Representation and Grid Interpolation, Continued: • Grid interpolation—SparseMatrix • Storage of global and local row and column indexing • Storage of matrix elements • Support for automatic decomposition of matrices, and generation of vector decompositions based on matrix decompositions

22. 5. Intercomponent communications and parallel data transfer: • Registry of components and their process pools—MCTWorld • Exchange between components of data decomposition descriptors • Parallel communications scheduler—the Router

23. Building the CCSM Next-Generation Coupler • The mct provides the “plumbing and wiring” support and compute core of the NGC • Other modules needed for the NGC— • A diagnostics / flux physics library • Component Model Interface routines

24. Component Model Interfaces What’s needed (first cut)— • Component model type identification • Ability to pack/unpack model data into AttrVect form • Conversion of model data decomposition descriptors into GlobalMap or GlobalSegMap form • Mechanisim to report the component model’s coordinate grid • A standardized scheme for identifying component model input/output field names

25. Component Model Interfaces The path to a software framework for climate system modeling— • Points 1-5 listed on the previous slide • Expansion of point 5 to include a standard mechanisim for reporting component model metadata regarding compatibility with other component models

26. Future development of the mct: • Support for on-line interpolation matrix element generation • Interface to SCRIP for horizontal • Vertical interpolation support • Performance optimization and inclusion of OpenMP • Upwards abstraction of data types • Support for other data storage and data decomposition types • Extension to support dynamically load balanced component models (but fixed process pool sizes) • Extension to support dynamically load balanced component models (dynamically varying process pool sizes)

27. Exciting, “within reach” applications of the mct: • Inclusion of other components in CCSM (e.g., atmospheric chemistry, biogeochemistry) • Use of globus-enabled MPI to construct coupled models with components residing on multiple platforms • Construction of a regional coupled model • Construction of non-earth-science “supermodels” (e.g., reactive flow) • Collaboration with the DOE Common Component Architecture (CCA) Forum