Demands on next generation dynamical solvers

1. Demands on next generation dynamical solvers Model and Data Hierarchies for Simulating and Understanding Climate Marco A. Giorgetta

2. (My) wish list for the dynamical solver • Conserves mass, tracer mass (and energy) • Numerical consistency between continuity and transport equation • Well behaved dynamics • Low numerical noise • Low numerical diffusion • Physically reasonable dispersion relationship • Accurate • Grid refinements (of different kinds) • Fast

3. Numerical consistency between continuity and transport equation

4. Example: ECHAM • A spectral transform dynamical core solving for: • Relative vorticity • Divergence • Temperature • Log(surface pressure) … using 3 time level “leap frog” time integration scheme • A hybrid “eta” vertical coordinate: • Pressure at interface between layers: p(k,t) = a x p(t) + b • Mass of air in a layer x g: dp(k,t) = p(k+1,t) - pi(k,t) • A flux form transport scheme • q, cloud water, cloud ice, (and tracers for chemistry or aerosols … using a 2 time level scheme  PROBLEM: tracer mass not conserved

5. Illustration: • ICON dynamical core + transport scheme • Triangular grid • Hydrostatic dynamics • Hybrid vertical “eta” coordinate • 2 time level semi-implicit time stepping • Flux form semi-Lagrangian transport scheme • Jablonowski-Williamson test • Initial state = zonally symmetric, but dynamically instable flow • Initial perturbation  Baroclinic wave develops over ~10 days • 4 Tracers, of which Q4(x,y,z,t=0) = 1

6. Daniel Reinert, DWD

7. Grid refinements (of different kinds)

8. Options/Questions • Grid refinement • static or dynamic/adaptive? • Re-distribute grid points or create/destroy grid points? • 2d or 3d? Boundary layer, troposphere, stratosphere, mesosphere • Single time integration scheme or recursive schemes? • Conservation properties? • Dynamical core • Adjust scheme to expected errors ( FE schemes) • Parameterizations: • Submodels: embedded dynamical models – “super-parameterizations” • Cost function • How to predict the need for refinement, and what for? Target/goal? • How to confine computational costs?

9. Generating the icosahedral triangular grid

10. Other kinds of grid refinements (A) Hexagon (B) re-distributing cells

11. Refining the grid Grid refinement Refinement by bi-section of triangle edges: 1 triangle  4 triangles 1 or more refined regions 1 or more refinement levels per region Leonidas Linardakis, MPI-M Two-way nesting Compute one time step on parent domain  dX/dt Interpolate the tendencies to the lateral boundary of the nested domain Perform 2 time steps(*) on the nested domain Feed back the increments (*) For more levels  Apply recursion  Numerical discontinuities!

12. Mountain induced Rossby waveVorticity at ~ 3 km MSL after 20 days of simulation 2000-m circular mountain at 30°N/90°E Initial condition: Zonal flow with maximum speed of 20 m/s Experiments: (A) Global with 140 km resolution (B) Global with 35 km resolution (C) As (A), but 2-step refined circular region with resolution of (B) (B) global resolution 35 km (A) global resolution 140 km (C) global res. 140 km, regional res. 35 km Günther Zängl, DWD

13. High Performance Computing

14. How to get faster: • Faster CPUs • Faster connections between CPU, memory and disk faster • Parallelization over more CPUs • CPUs share memory • CPUs have their own memory • Modify code design to account for architecture of CPUs • Scalar/vector CPUs • Sizes of intermediate, fast access memories (“Caches”) • The past was dominated by improved CPUs • The future will be dominated by more CPUs

15. Parallelzation • Distribute the work to many CPUs • Works well for local computations : cells, columns, (levels), … • Works badly for non-local task: integrals, global organization, … • Serial and parallel sections in a code • On 1 CPU: • On 4 CPUs:

16.  Amdahl’s law The serial fraction of work limits the speedup • P = fraction of the workthat can be parallelized • 1-P = remainder, whichcannot be parallelized • S = 1/(1-P) =maximum speedupfor N  ∞ • (Wikipedia) The serial fraction of work limits the maximum speedup!

17. For illustration: Computer at DKRZ • IBM Power6 CPUs • 250 nodes • 16 CPUs/node  total = 4000 CPUs • 2 cores/CPU  total = 8000 cores • 2 floating point units/core • 64 parallel processes/node • ECHAM GCM at ~1° resolution • Scales “well” up to 20 nodes = 640 cores (with parallel I/O) • Problem: Spectral transform method used for dynamical coreRequires transformations between spherical harmonics and grid point fields  global data exchange, transpositions. • Future: ~105 cores  New model necessary

18. Strategies • Select numerical scheme, which is • Sufficiently accurate with respect to your problem • Computationally efficient • Fast on single CPUs • Minimize global data exchange (Transformations, “fixers”, I/O) • Find optimal way to distribute work • Practical issues: • Optimization the code for the main computer platform • Account for strengths/weaknesses of available compilers • Avoid “tricks” which will stop the code to work on other platforms • Optimize first the most expensive parts

19. Other problems in HPC • Data storage: • Disk capacities grow less than computing power • Bandwidth between computer and storage system • ESMs can produce HUGE amounts of data • Finite lifetime of disks or tapes  backups or re-computing? • Data accessibility: • Bandwidth between disks/tapes and post-processing computer • Post-processing software must be parallelized • Data description • Documentation of model, experimental setup, formats etc. • Climate models are no longer a driver for the HPC development

20. END