Earth System Modeling Framework

Earth System Modeling Framework ESMF Introductory Tutorial ESMF v 3.0.1 http://www.esmf.ucar.edu

Introduction and Orientation http://www.esmf.ucar.edu

Outline • Administrivia • Background • Building ESMF • HPC terminology • How to create an ESMF component • Coupling • Setting up a parallel environment • ESMF infrastructure: Data structures • ESMF infrastructure: Utilities

Rough Timeline • [8:30-9:00] Introduction and orientation • [9:00-10:30] Building and linking exercise • [10:30-10:45] Break • [10:45-noon] Terminology, superstructure, coupling exercise • [noon-1 pm] Lunch on your own • [1:00-2:20] Parallel environment, data structures, methods, and code review • [2:20-2:35] Break • [2:35-3:00] Utilities

Administrivia • Bathrooms down the hall • Cafeteria in next building over • Refreshments in the hall • No stupid questions, ask anytime • You can tell us to speed up or slow down • During exercises let us know if you are having problems or don’t understand

What is ESMF? • What is a framework? • Framework vs. library • A toolbox for modelers • Turn model codes into components • Couple model components • Provide code infrastructure • Time management • Log management • Etc.

Superstructure vs. Infrastructure • Superstructure: The means by which models are converted into components and coupled • Infrastructure: Tools that modelers can use within their own code • Data structures: Array, Field, etc. • Utilities: TimeMgr, LogErr, etc. • These are independent, but work together

ESMF Structure GridComp Land, ocean, atm, … model CplComp Xfers between GridComps State Data imported or exported Superstructure Infrastructure Regrid Computes interp weights Bundle Collection of fields Field Physical field, e.g. pressure Grid LogRect, Unstruct, etc. PhysGrid Math description DistGrid Grid decomposition F90 Array Hybrid F90/C++ arrays DELayout Communications Route Stores comm paths C++ Utilities Virtual Machine, TimeMgr, LogErr, IO, ConfigAttr, Base etc. Communications Data

What’s New in ESMF 3 • Simplified build and link • Public DistGrid - stores grid topology • New Array Class – extensions to native arrays • Regridding w/ ESMF_ArraySparseMatMul() • User provided interpolation weights • Parallel sparse matrix multiply

Support • Mailing lists to join: • esmf_jst@ucar.edu: Technical discussion • esmf_info@ucar.edu: Quarterly newsletter • esmf_community@ucar.edu: Community announcements • Location to join: http://www.esmf.ucar.edu > Users • Support: • Send email to: esmf_support@ucar.edu

Web • Main site: http://www.esmf.ucar.edu

Reference Manual • Go to http://www.esmf.ucar.edu/ • Click Download tab • Click View All Releases link (left nav bar) • Scroll down to v3_0_1 • Click on the HTML version of the Reference Manual in the table

Downloading • Go to http://www.esmf.ucar.edu/ • Click Download tab • Latest public release here • Click View All Releases link (left nav bar) • Scroll down to v3.0.1 • Click to download and accept license • Alternate: Go to http://cdp.ucar.edu • Must register and join ESMF group • All releases are available • Source is a gzipped tar file - must be unpacked

Feature Prioritization • Issues and feature requests arrive via esmf_support@ucar.edu • SourceForge tracker ticket is created • CRB reviews issues and establishes release schedule: • http://www.esmf.ucar.edu/plans/releases/index.shtml • Prioritization can be influenced by participating in JST telecons

Future Development • New Grid development • Support for structured grids • Support for unstructured meshes. • Automated regridding • C framework interfaces

Questions?

Building ESMFHands-On

Section Outline • Requirements to build • Set environment variables • Build library • Common build problems • Directory structure • Build and run tests

Requirements to Build • Fortran 90 compiler (g95, ifort, …) • C++ compiler (gcc, icc, …) • MPI libraries (MPICH, LAMMPI, ...) • gnu’s make gmake (not make) • prompt$ make -v • indicates type • Perl (for running test scripts)

Environment Variables: Review • Typical values for Linux machines • ESMF_DIR: Top level ESMF directory path • ESMF_BOPT: ‘g’ for debug, ‘O{0,1,2,…}’ optimize • ESMF_COMM: mpich, mpich2, lam, openmpi, intelmpi, or mpiuni for serial mode. • ESMF_COMPILER: intel, xlf, absoft,lahey, pgi, pathscale, g95 • ESMF_ABI: (Application binary interface) System pointer word size/memory model; 32 or 64 • See the README file and “Environment Variables” in the “Quick Start” section of the User’s Guide for more info

Required Environment Variables • Environment variables must be set prior to building the source. • export ESMF_DIR=absolute_path • export ESMF_BOPT=g • export ESMF_COMM=mpich2 • export ESMF_COMPILER=g95 • Check the settings with • make info

Building the Framework • Download the tarball (already done for tutorial) • Unpack and build the framework • prompt$ gunzip esmf_3_0_1_src.tar.gz • prompt$ tar -xvf esmf_3_0_1_src.tar • Set environment variables • prompt$ more set* • prompt$ export ESMF_DIR=absolute_path • prompt$ export ESMF_BOPT=g • prompt$ export ESMF_COMM=mpich2 • prompt$ export ESMF_COMPILER=g95 • Build: prompt$ cd ~/esmf; make

Review Build Commands • Clobber any previous build • make clobber • Build the framework • make • Display build settings • make info • NOTE about gnumake: On many Linux systems gnumake or gmake is renamed make. Typically this is not the case on non-Linux platforms.

Common Build Problems • Old versions of required software • gcc • gnumake • Environment variable settings • Library paths • 32 bit machines set to 64 bit • Software install • Compiler build options • MPI build options

Modify Build Files • prompt$ cd ~/esmf/build_config • prompt$ ls • Linux.absoftintel.default/ • Linux.pgigcc.default/ • Linux.g95.default/ • Linux.gfortran.default/ • Linux.xlf.default/ • Linux.intel.default/ • See the “Porting to New Platforms” section of the User’s Guide

Directory Structure (1 of 2) • Change to the framework directory • prompt$ cd ~/esmf/ • README: Overview ESMF of build & run • application/: Example of a standalone application code • build/: Make helper file common.mk • build_config/: Config file for building on different platforms • lib/: Location of ESMF library (esmf.mk & libesmf.a)

Directory Structure (2 of 2) • makefile: Top level makefile for building and running • mod/: Shared modules • quick_start/: Toy example of a coupled model • src/: Source code • demo/ • Infrastructure/ • Superstructure/ • system_tests/

Starting MPI • Required to run • Some flavors of MPI, such as LAM & MPICH2, require starting an external process manager before running programs. • mpdboot MPICH2 • Run the something here!!!!! • run_something • Halt MPI • mpdallexit MPICH2

Verify the Build • Build the unit tests • prompt$ make build_unit_tests • Run the unit tests • prompt$ mpdboot • prompt$ make run_unit_tests

What Did We Learn? • Environment variables are key • gnumake info can helpful in debugging • The library build takes time • Where to find things in the source directories • Running the tests can confirm the build

Exercise: Installing ESMF and Building and Linking to a Stand-Alone Application

ESMF Installation • Extracts the minimum library and helper files needed to link any stand-alone application to the framework • Allows these files to be placed in a public space • Eliminates need for keeping the entire source tree

ESMF Installation • prompt$ make info • Examine the installation environment variables • Set installation environment variables • ESMF_INSTALL_PREFIX=~/ESMF301 • ESMF_INSTALL_MODDIR=mod • ESMF_INSTALL_LIBDIR=lib • prompt$ more stand* • prompt$ make install • prompt$ make installcheck • See the “Building and Installing” section of the User’s Guide for more info

The Stand-Alone Code • prompt$ mkdir ~/exercise • prompt$ chmod 755 -R ~/esmf • prompt$ cd ~/esmf/application • prompt$ ls • Makefile • esmf_application.f90 • prompt$ cp * ~/exercise • prompt$ cd ~/exercise • prompt$ more Makefile

Setting up the Makefile(1 of 3) • Step 1: Completing the Makefile path • The Makefile requires a path to an esmf.mk • There can be multiple versions if you have built with different flags • Different versions represent different build choices • Debug or Optimize • Compiler choice • MPI choice • Step 2: • prompt$ cd ~/esmf/lib/ • prompt$ ls • Explore the directories created by your build choices

Setting up the Makefile(2 of 3) • Step 3: Choose the path to esmf.mk consistent with the desired system configuration • Example path: • ~/esmf/lib/libg/Darwin.absoft.32.lam.default/ • libg => set up in debug mode • Darwin => operating system • Absoft => compiler • 32 => ESMF_ABI • lam => MPI flavor • What is your path?

Setting up the Makefile(3 of 3) • Step 4: prompt$cd ~/ESMF301/lib • Step 5: prompt$ ls • Notice the alternative path to esmf.mk • Save this path • Step 6: prompt$ cd ~/exercise/ • Step 7: Open the Makefile in an editor • Replace the dummy path with the captured path, to end of the variable ESMF_INSTALL_LIBDIR_ABSPATH on line 14 • Step 8: Save and exit

Building & Running the Application • Build • prompt$ cd ~/exercise • prompt$ make • Running • prompt$ mpdboot • prompt$ mpirun -np 4 esmf_application • prompt$ mpdallexit

What Did We Learn? • All we need to link to the ESMF library with a stand-alone application is the appropriate esmf.mk file • We don’t need the environment variables to build and link to the the stand-alone application once the framework is built.

Questions?

HPC Terminology

Serial vs. Parallel • Future computing platforms will consist of multiple processors • Often don’t share memory • Can have multiple threads of execution per processor • MPI & OpenMP are the standards • Programs can run in a serial fashion, w/ one thread or, • Parallel using multiple threads

PETs: Persistent Execution Thread • We call threads PETs • In this introductory tutorial, we’ll stick with one PET per processor • Serial applications run on one PET, parallel applications run on multiple PETs

Decomposition Elements (DEs) • A data decomposition is represented as a set of Decomposition Elements (DEs) • In many applications there is one DE per PET

Simple Decomposition Illustration

Sequential Execution • Components run one after another on the same set of PETs

Earth System Modeling Framework