Improved Scalability of the Finite-Volume Dynamical Core

1. Improved Scalability of theFinite-Volume Dynamical Core The Cubed-Sphere and 2-dimensional (XY) domain decomposition William M. Putman Special thanks to Shian-Jiann Lin (GFDL) V. Balaji (GFDL) Tom Clune (SIVO NASA/GSFC) NASA/GSFC Software Integration and Visualization Office NASA/GSFC Software Integration and Visualization Office

2. ObjectiveHighly scalable fvcore • Scalability of the Finite-Volume dynamical core to 10,000s - 100,000s of processors • 2-Dimensional horizontal (XY) domain decomposition • Quasi-Uniform grid allows Purely Flux-Form dycore • No semi-lagrangian extension • No polar filters • Stepping-stone toward global non-hydrostatic cloud-resolving atmospheric model at 4-km to 1-km horizontal resolution (Lin, GFDL) NASA/GSFC Software Integration and Visualization Office

3. IntroductionThe fvcore and the cubed-sphere • The FV dycore • Developed at NASA during the 1990’s (Lin & Rood, 1990s) • An Integral component of the GEOS atmospheric model line at NASA/GSFC • Lat-Lon implementations exist within the Global modeling efforts at NASA, NCAR, NOAA-GFDL • GEOS-4 and GFDL implementations • 1-D domain decomposition with MPI in latitudinal direction OpenMP in vertical • GEOS-5 and NCAR CAM implementations • 2-D domain decomposition in latitude and vertical directions • The Cubed-Sphere • Potential Scalability to 100,000s of processors • Ideal for 2-dimensional XY domain decomposition • Quasi-Uniform mapping of the cube to a sphere • Gnomonic (Sadourny, 1972) • Conformal (Rancic & Purser, 1996) • Elliptic Solvers • Can share a common code base with lat-lon implementation NASA/GSFC Software Integration and Visualization Office

4. MotivationScaling to 100,000s of Processors • Why? • Climate (capacity computing) • 100-km to 50-km resolution • Long integration (100+ years) • 100s of chemical constituents • Many scenarios - Faster turnaround • Advanced Physics and Super-Parameterization • Hi-Res Cloud resolving model within each grid cell • Thirst for more processors • Weather (capability computing) • High Resolution NWP • 25-km to 50-km at present • 5-km to 10-km within a decade • Global Cloud resolving • Non-Hydrostatic Convective-Parameterization free (explicit cloud micro-physics) NASA/GSFC Software Integration and Visualization Office

5. MotivationScaling to 100,000s of Processors • Current Limitations • Finite-Volume dycore • The FV algorithm of Lin and Rood (1990s) • Built on Lat-Lon grid • Convergence of longitudes at poles • Polar Filters • Flux-Form with Semi-lagrangian extension • Non-local communication for XY domain-decomp • Limited scalability due to 1-D domain-decomp • YZ - decomp option • Requires XY to YZ transformation for column physics NASA/GSFC Software Integration and Visualization Office

6. MotivationScaling to 100,000s of Processors • How do we get there? • Quasi-Uniform Grids • Cubed-Sphere grid • Geodesic grids • 2D XY domain-decomposition • Exposes additional parallelism over Lat-Lon • Potential scalability of Cubed-Sphere • Scalable communication • Barriers within the FV algorithm • Orthogonality requirements • Vector interpolation C-D grid scheme • Vector Components on the corners • The algorithm needs them • Special treatment on cubed-sphere • Vorticity at corner points • Other efforts • The high-order Discontinuous Galerkin on Cubed-Sphere at NCAR (HOMME) • Finite-Volume methods on geodesic grids at CSU NASA/GSFC Software Integration and Visualization Office

7. ApproachThe FV dycore on the Cubed-Sphere • The choice of Grid • Gnomonic, Conformal, Elliptic-Solvers… • Characteristics of computationally efficient grids • Minimum Grid length • Orthogonal • Conformal • preserves the angles between intersecting curves on adjoining faces • Differentiable to any order everywhere except singularities • Aspect Ratio (smoothness) • Conformal versus Elliptic-Solvers • Orthogonality versus Minimum Grid-Length NASA/GSFC Software Integration and Visualization Office

8. ApproachThe FV dycore on the Cubed-Sphere • FV algorithm modifications • Generalized Software • Work on a generic Cartesian patch • Replace lat/lon geometric factors with areas and grid lengths • Special handling of poles and cyclic BCs (lat-lon) • Special handling of corners (cubed-sphere) • Vector interpolation • Divergence Conserving • Adjustment factor around corners • D-grid kinetic energy formulation • Upwind Biased • Advection of D-Winds using B-Winds • B-Winds ambiguously defined at corners • Vorticity at corner points • Circulation theorem • C-Grid Winds • 2-D XY domain-decomposition • Uses Mosaics within MPP from GFDL • Block decomposition within each face NASA/GSFC Software Integration and Visualization Office

9. ResultsShallow Water Dynamics and Scalability • Shallow Water Equations • What are they? • Mass conservation • Vector invariant momentum equation • Why do we use them? • standard tests cases (Williamson et al) • comprehensive evaluation of the numerical implementation of the dynamical core • Widely accepted as initial evaluation of numerical schemes prior full three-dimensional baroclinic implementation • Initial scaling results • 2D XY domain-decomposition • SGI Altix (512 processors) NASA/GSFC Software Integration and Visualization Office

10. Test Case 1 • 2D Advection of a cosine bell • Advection Oriented over Corners • Height error after 1 revolution (12-days) • Mass conservation

11. Test Case 2 • Zonal Geostrophically Balanced Flow • Steady State Solution • Flow Oriented over Corners

12. Test Case 2 • Zonal Geostrophically Balanced Flow • Reveals excitation of gravity waves around corners due to geostrophic inbalances in the initial state and D- to C- Grid vector inperpolation • Rossby wave mode (zonal wave number-4) excited due to variation in resolution near corners • Error convergence with increased resolution displayed (2nd order convergence)

13. Test Case 5 • Zonal Geostrophically Balanced Flow over an isolated mountain • Excitation of multiple waves due to presence of mountain • Tests energy properties during conversion between potential and kinetic energy

14. Test Case 6 • Rossby Wave (Wavenumber-4) • 60-day Integration

15. Strong Scalability -- where the problem size is fixed and the processor count expands • 2-Dimensional X-Y Domain decomposition • Pure MPI implementation (Can also be hybrid MPI-OpenMP in 3D dycore) • Run on SGI Altix Cubed-Sphere Scalability

16. Weak Scalability – where the problem size and processor count expand • 2-Dimensional X-Y Domain decomposition • Pure MPI implementation (Can also be hybrid MPI-OpenMP in 3D dycore) • Run on SGI Altix Cubed-Sphere Scalability

17. Summary and Future • Targeting a global atmospheric model for climate and weather scalable to 100,000s of processors • Current status • A 2-dimensional shallow water model with the finite-volume dynamics on the Cubed-Sphere • 2D XY domain-decomposition • Demonstrating scalability beyond the limits of the 1D domain-decomp on lat-lon • What’s next? • Implementation within the 3-dimensional baroclinic framework • Thermodynamic equation • Lagrangian vertical coordinate • Pressure gradiant calculations • Full physics implementation • Scaling studies at 1,000s - 10,000s of CPUs NASA/GSFC Software Integration and Visualization Office