An Object-Oriented Software Framework for Building Parallel Navier-Stokes Solvers

1. An Object-Oriented Software Framework for Building Parallel Navier-Stokes Solvers Xing Cai Hans Petter Langtangen Otto Munthe University of Oslo

2. Outline of the Talk • Mathematical model for N-S equations • Different numerical methods • An O-O framework in Diffpack • Parallelization • A fast FEM N-S solver • A test case

3. Mathematical Model Incompressible Navier-Stokes equations:

4. Different Solution Methods • Fully implicit methods • Artificial compressibility • Operator splitting (discrete & continuous) • Solenoidal approaches • Pressure filtering Many choices and possibilities!

5. An Observation • Most solution processes have sub-steps • Each sub-step solves one/several PDEs • Many PDEs in the sub-steps are of the same type • Re-usable simulators • Small adjustments needed

6. An O-O Framework • Goal: Flexible and extensible simulators • Take advantages of OO programming • Unified implementation of different base PDE solvers • Unified organization of common data • Separate administration of problem specific data (flow conditions)

7. NsPrms SomeSolver NsCase SomeCase NsBase Implementation Structure Use of OOP only on high administration level Fortran construct and c-array at low-level

8. Diffpack • O-O software environment for scientific computation • Rich collection of PDE solution components - portable, flexible, extensible • http://www.nobjects.com • H.P.Langtangen, Computational Partial Differential Equations, Springer 1999

9. Straightforward Parallelization • Develop a sequential simulator, without paying attention to parallelism • Follow the Diffpack coding standards • Use add-on libraries for parallelization specific functionalities • Add a few new statements for transformation to a parallel simulator

10. Linear-algebra-level Approach • Parallelize matrix/vector operations • inner-product of two vectors • matrix-vector product • preconditioning - block contribution from subgrids Keeps original sequential Diffpack libraries almost intact needs inter-processor communication functionalities • Easy to use • access to all existing Diffpack iterative methods, preconditioners and convergence monitors • need only to add a few lines of new code • arbitrary choice of number of procs at run-time

11. Work Load Distribution • Through grid partition • Need good load balance • Flexibility & extensibility • Global grid -> a set of subgrids • Arbitrary number of procs determined at run-time • Non-overlapping partition • Controllable addition of overlap (if desired) • An existing set of subgrids (input from files)

12. Partition example

13. An Add-on Parallelization Library • Grid partition administration • High-level inter-processor communication functionalities (hidden MPI) • class GridPartAdm • void GridPartAdm::prepareSubgrids() • void GridPartAdm::prepareCommunication() • void GridPartAdm::updateGlobalValues() • void GridPartAdm::matvec • void GridPartAdm::innerProd • void GridPartAdm::norm Fully portable (Origin 2000, IBM SP2, HP V2500...)

14. A Simple Coding Example // ... #ifdef PARALLEL_CODE adm->scan (menu); adm->prepareSubgrids (); adm->prepareCommunication (); lineq->attachCommAdm (*adm); #endif lineq->solve (); // ... set subdomain list = DEFAULT set global grid = grid1.file set partition-algorithm = METIS set number of overlaps = 0

15. LAL Parallelization Example Highly unstructured grid Discontinuity in the coefficient K

16. Measurements for example one 130,561 degrees of freedom Overlapping subgrids BiCGStab + (block) RILU prec.

17. A Fast FEM N-S Solver • Operator splitting • Calculation of an intermediate velocity

18. A Fast FEM N-S Solver • Solution of a Poisson Equation • Correction of the intermediate velocity

19. NsCase NsSolver Pressure Explicit Predictor Corrector Case1 FEM Case2 NsBase O-O Implementation

20. Test Case: Vortex-Shedding

21. Simulation Snapshots Pressure

22. Simulation Snapshots Pressure

23. Animated Pressure Field

24. Simulation Snapshots Velocity

25. Simulation Snapshots Velocity

26. Animated Velocity Field

27. Some CPU-Measurements

28. Parallelization Approach 2: DD • Parallelization of multilevel methods • One subdomain is assigned with a sequential simulator • A generic implementation framework • Systematic and flexible • O-O programming enables extensive code reuse • Easy to incorporate multilevel algorithm modification • Different grid types, local solution methods etc. on different subdomains

29. Parallel DD Application One 2D pressure equation in reservoir simulation

30. Parallel DD Application Two 3D Poisson equation in water wave simulation

31. Parallel efficiency • Fixed number of subdomains M=16. • Subdomain grids from partition of a global 41x41x41 grid. • Simulation over 32 time steps. • DD as preconditioner of CG for the Laplace eq. • Multigrid V-cycle as subdomain solver.

32. Summary • O-O implementation framework for N-S solvers • Flexible & user-friendly parallelization at the level of linear algebra • Parallelization at the level of DD