“High Performance Computing and Simulation Symposium 2008” Ottawa, Canada, April 14-16, 2008

1. Solution of the Implicit Formulation of High Order Diffusion for the Canadian Atmospheric GEM Model “High Performance Computing and Simulation Symposium 2008” Ottawa, Canada, April 14-16, 2008 Abdessamad Qaddouri & Vivian Lee Atmospheric Science & Technology

2. Outline • Introduction of GEM Model • High order Diffusion equation and solution • Parallelization of the solution • Numerical performance Tests • Conclusion

3. Numerical Weather Prediction (NWP) • Physics • Applied Mathematics • Real-time applications • Computers at Canadian Meteorological centre (CMC)

4. Forecast lead time 250-400 km resolution (4 times per year) 250 km resolution (twice per month) 15 km resolution (twice per day) 2.5 km resolution(once per day) 35 km resolution(once per day) 100 km resolution (once per day) Statistical (4 times per year) 0 1 2 5 10 30 90 365 (days) deterministic forecasts probabilistic forecasts empirical forecasts

5. Limited Area Variable Uniform Rotated 35km=800x600x58 2.5km=672x494x58 15km= 574x641x58

6. Hydrostatic Model • Horizontal motion (momentum) • Thermodynamics, hydrostatic and state • Continuity and boundary conditions

7. Schematic for Semi lagrangian implicit Method used for the integration of GEM Model Discretization Trajectory Nonlinear Iterations Diffusion on specific fields

8. Horizontal High order Diffusion • Horizontal prognostic field • Damping rate Damping rate Wave-length

9. Horizontal High order Diffusion… • Horizontal prognostic field • Implicit Discretization

10. Horizontal High order Diffusion … • Del 4 Horizontal Diffusion • Spatial Discretization

11. Spatial disretization

12. Horizontal High order Diffusion … • Fast Direct Solution • Projection

13. Horizontal High order Diffusion … • Direct Solution • Matrix Form

14. Horizontal High order Diffusion … • Block Tri-diagonal problem solution • Solution

15. Summary of the algorithm • Analysis of the right hand side (FFT or MMM) • Solution of (Nk*Ni) tri-diagonal Problems • Synthesis of the solution (FFT or MMM)

16. A Parallel algorithm • Global Transposition (Ni/P,Nj/Q,Nk) (Nj/Q,Nk/P,Ni) • Analysis of the right hand side • Global Transposition (Nj/Q,Nk/P,Ni) (Nk/P,Ni/Q,Nj) • Solution of the block tridiagonal problems • Global Transposition (Nk/P,Ni/Q,Nj) (Nj/Q,Nk/P,Ni) • Synthesis of the solution • Global Transposition (Nj/Q,Nk/P,Ni) (Ni/P,Nj/Q,Nk)

17. 35km mesoglobal run At 72hr forecast U component without diffusion U component with DEL 6 diffusion

18. Table 1. Breakdown of timings in the major components of the Canadian 35Km mesoglobal operational model for an integration of 72 hours on 12 nodes (2 x 24 x 4)

19. Table 2. MPI test runs for 35km mesoglobal (OpenMP=1);the number of calls to the diffusion is 964 times

20. Table 3. MPI test runs for 17 Km mesoglobal (OpenMP=1); the number of calls to the diffusion is 964 times.

21. MPI Relative Speedup • 35km Mesoglobal FFT 17km Mesoglobal FFT

22. Table 4. OpenMP test runs for 35Km mesoglobal configured (1 x 16 x OpenMP) using FFT: the number of calls to the diffusion is 964 times.

23. Table 5. OpenMP test runs for 35Km mesoglobal configured(1 x 16 x OpenMP) using Matrix multiplication: the number of calls to the diffusion is 1084 times.

24. OpenMP relative Speedup • 35km Mesoglobal FFT 35km Mesoglobal MXM

25. Conclusion • An efficient implementation of the parallel Fast Direct Solution for the implicit formulation of horizontal diffusion problem • Comparison with iterative methods like preconditioned Krylov methods.

26. Thank You! Merci!