The –COSMO Project Overview, Status and Plans

1. The –COSMO ProjectOverview, Status and Plans Oliver Fuhrer, MeteoSwiss Project Team: Anne, Christoph, Christoph, Daniel, David, Dominik, Florian, Isabelle, Jean-Guillaume, Jürg, Katharina, Lukas, Matthew, Men, Neil, Oliver, Thomas, Thomas, Tim, Tobias, Ulrich, Will, Xavier

2. The Inititative • HP2C = High Performance High Productivity Computing • Overarching GoalPrepare computational sciences to make effective use of next generation supercomputers • Specific GoalEmerge with several high-impact scientific applications that scale and run efficiently on leadership computing platforms in 2012/13 timeframe • Websitehttp://www.hp2c.ch/ Courtesy: Thomas Schulthess

3. The Approach • Start with a real and challenging scientific problem • Take the best methods known in the field • Understand their limitations on today’s emerging hardware • Algorithmic reengineering to better map method to hardware • Take aggressive view on rewriting codes – don’t hesitate to rewrite major portions or even starting from scratch • Pay close attention to hardware developments Courtesy: Thomas Schulthess

4. Project Organization • BigDFT - Large scale Density Functional Electronic Structure Calculations in a Systematic Wavelet Basis Set (Prof. Stefan Goedecker, University of Basel) • Cardiovascular - HPC for Cardiovascular System Simulations (Prof. Alfio Quarteroni, EPF Lausanne) • COSMO-CCLM - Regional Climate and Weather Modeling on the Next Generations High-Performance Computers: Towards Cloud-Resolving Simulations (Dr. Isabelle Bey, ETH Zurich) • Cosmology - Computational Cosmology on the Petascale (Prof. George Lake, Uni Zürich) • CP2K - New Frontiers in ab initio Molecular Dynamics (Prof. Juerg Hutter, Uni Zürich) • Ear Modeling - Numerical Modeling of the Ear: Towards the Building of new Hearing Devices (Prof. Bastien Chopard, University of Geneva) • Gyrokinetic - Advanced Gyrokinetic Numerical Simulations of Turbulence in Fusion Plasmas (Prof. Laurent Villard, EPF Lausanne) • MAQUIS - Modern Algorithms for Quantum Interacting Systems (Prof. Thierry Giamarchi, University of Geneva) • Neanderthal Extinction - Individual-based Modeling of Humans under climate Stress (Prof. C. P. E. Zollikofer, University of Zurich) • Petaquake - Large-Scale Parallel Nonlinear Optimization for High Resolution 3D-Seismic Imaging (Prof. Olaf Schenk, Uni Basel) • Selectome - Selectome, looking for Darwinian Evolution in the Tree of Life (Prof. Marc Robinson-Rechavi, University of Lausanne) • Supernova - Productive 3D Models of Stellar Explosions (Prof. Matthias Liebendörfer, University of Basel) Finite difference methods on structured grids

5. The COSMO Model Finite differences on structured Grid Timestepping Setup Input Physics Dynamics Assimilation Boundary Conditions Diagnostics Output

6. Lines vs. Runtime ~250k lines of Fortran 90 code % Code Lines % Runtime active

7. Prototype Code 1 (Dynamics) • Solve PDE of diffusion equation (4 FLOPS, 3 load/stores) } } halo-update

8. Prototype Code 2 (Physics) • Higher computational density (136 FLOPs, 3 load/stores) no comm

9. Memory Scaling • Example: COSMO-2 “like” on Cray-XT4 • Problem size: 102 x 102 x 60 gridpoints on 60 cores • Keep number of cores constant, vary number of cores/node used P P P Relative Runtime (4 cores = 100%)

10. The – COSMO Project Task 1 Prepare COSMO for convection resolving IPCC runs Task 2 Incremental optimization of existing code Understanding of workflow, benchmarking Hybrid parallelization MPI/OpenMP Parallel I/O Talks: Anne+Jean-Guillaume, Neil, Matt Task 3 Fundamental redesign of COSMO code Redesign data architecture for optimal cache usage Rewrite dynamical core Investigate shift to GPUs Talks: Tobias, Xavier, Will Duration June 2010 – December 2012

11. GPU Implementation • GPUs are very attractive… • higher memory bandwidth • higher FLOPS per Watt • low cost • But… • need to consider the whole workflow • COSMO is a community code and will always (?) have to run on a commodity cluster • the code should stay easy to understand and modify • New project proposal (OPCODE) for a GPU implementation of MeteoSwiss operational suite submitted

12. “Other” Applications Apart from plain vanilla COSMO runs, COSMO is also run in “exotic” configurations Examples COSMO-CLM NetCDF input/output COSMO-ART gas-phase and aerosol chemistry COSMO-M7 aerosol-radiation feedback COSMO-CLM2 land surface scheme … These configurations may potentially change “everything” Example: COSMO-ART

13. COSMO-ART Additional 58 gas and 105 aerosol variables (normally 11). Of which 56 and 77 are transported and diffused. COSMO (6 min) COSMO-ART (105 min) 24h simulation Cray XT4 (dole) 182 x 170 x 40 gridpoints 10 x 11 cores

14. General Conclusions • POMPA is the platform to bring the HP2C activities into the COSMO Consortium and collaborate/coordinate activities within COSMO • Bringing domain scientists, computational scientists and computer engineers together takes time, but it is worth the effort • Take a look at https://hpcforge.org/projects/cclm-dev/ Thank you!