Parallel Preconditioners for the Incompressible Navier-Stokes Equations

1. Parallel Preconditioners for the Incompressible Navier-Stokes Equations Robert Shuttleworth Applied Math & Scientific Computation (AMSC) University of Maryland AMSC Candidacy Presentation

2. Outline • Background • Incompressible Navier-Stokes Equations • Discretization/Linearization • Preconditioning the N-S Equations • General Preconditioners • N-S Problem Specific • Pressure Correction Methods • Pressure Convection-Diffusion • High Performance Computing Issues • Preliminary Results • Lid driven cavity problem • Flow over a diamond obstruction • Conclusions AMSC Candidacy Presentation

3. Motivation/Focus • Motivation: Efficient and robust solution of steady state and transient flow problems • Develop fully implicit solution methods to the incompressible Navier-Stokes • Solving the linear systems that arise can take upwards of 70% of the CPU time of a given simulation • Linear Solvers: Operator Based Block Preconditioning • Focus: Adapt block preconditioners to the linear subproblems that arise in realistic fluid flow problems AMSC Candidacy Presentation

4. Introduction Given the Incompressible Navier-Stokes Equations: Nonlinear Term: Oseen: Newton: Jacobian of Momentum Eq. Discretization and Linearization: AMSC Candidacy Presentation

5. Discretization and Linearization AMSC Candidacy Presentation

6. Complete Algorithm u(0) = initial condition or initial guess p(0) = initial condition or initial guess for m = 1:Ntimesteps u(m) = u(m-1) , p(m) = p(m-1) while || F (u(m) ,u(m-1) ,p(m) ,u(m) ) || > nonlin ulag = u(m) /* */ /* Set up linear subproblem */ /* */ /* corresponding to F (u(m) ,u(m-1) ,p(m) ,ulag ) = 0. */ Iterate on A u(m) = b until || rk || /|| r0 || < saddle Block Precondition time loop nonlinear loop linear solver AMSC Candidacy Presentation

7. General Preconditioning Premise • Preconditioning needed in solution of any large scale PDE • Bottleneck of solving N-S is the iterative solution of the linear systems • Given: • Preconditioning speeds up convergence by improving the spectral properties of a matrix “Good” “Cheaper” AMSC Candidacy Presentation

8. Types of Preconditioners • General (Algebraic) Preconditioners • Incomplete LU Factorization (ILU) • Sparse Approximate Inverses (SPAI) • Multigrid • Domain Decomposition • N-S Problem Specific Preconditioners • Pressure Correction • Pressure Convection-Diffusion AMSC Candidacy Presentation

9. Incomplete LU Factorization (ILU) • Factoring a sparse matrix by Gaussian Elimination generates fill-in. So, the L and U factors are less sparse than the original matrix. • By ignoring the fill-in that occurs within a certain tolerance, approximate factors to L and U are available. • Advantages – simple to implement, inexpensive, good for certain problems • Disadvantages – potential instabilities, lack of scalability, not good for CFD applications, and difficulty in parallelization AMSC Candidacy Presentation

10. Block Preconditioners • Discretization • Consider: • Optimal preconditioner is when X is the Schur Complement, • Question: How to approximate the Schur complement? AMSC Candidacy Presentation

11. Pressure Correction – (LD)U So, we can apply a preconditioner to the saddle point matrix of the form: AMSC Candidacy Presentation

12. Pressure Correction • SIMPLER: Projection Matrix: Enforces Incompressibility AMSC Candidacy Presentation

13. Pressure Correction • Advantages • Used in both transient and steady state • Easy to implement and parallelize • Disadvantages • Slower convergence – coupling of physics is violated • Choosing a relaxation parameter • Inefficient for convection dominated flows AMSC Candidacy Presentation

14. Pressure Convection-Diffusion – L(DU) Therefore, a right oriented preconditioner can be applied to this problem: AMSC Candidacy Presentation

15. Pressure Convection Diffusion - Fp Suppose: Suppose the velocity and pressure convection-diffusion terms commute with one another: Then, AMSC Candidacy Presentation

16. Pressure Convection Diffusion - BFB AMSC Candidacy Presentation

17. Pressure Convection-Diffusion • Advantages • Insensitive to mesh size, time step, and CFL number • Minor Reynolds number dependence • Solves coupled system • Disadvantages • Applications do not supply Fp • Designed for Oseen iterations • Equations for Stabilized FEM are not developed • Boundary Conditions AMSC Candidacy Presentation

18. Packages: Time Loop Component Methods Package Nonlinear Loop Fluid Flow Finite Element MPSalsa (Epetra) Linear Solver Nonlinear Solver Newton-Krylov Methods NOX Block Precond Linear Solver GMRESR Aztec00 (Epetra, TSF) block precondition Meros (TSF) End NonLin Loop End Time Loop F-1 : GMRES/AMG X-1 : CG/AMG Aztec00, ML Epetra AMSC Candidacy Presentation

19. Epetra – Sparse Matrix Package • Facilitates sparse matrix construction on both parallel and serial machines • Compressed Row Storage (CRS) • Double precision nonzero values are stored in contiguous memory locations • Builds a map (graph) – an array of integers corresponding to nonzero row/column entries • Rows are stored in consecutive order AMSC Candidacy Presentation

20. Epetra - Sparse Matrix Library • Serial – interfaces the BLAS • Parallel - handles distributed matrix details • Local versus global indices • Global Map (graph) details which processor owns which entries • Local Map (graph) details how local data (and ghosts) is represented • Matrix-vector products • Wrappers to Message Passing Interface (MPI) (Note: The global map is normally determined by the user or other library.) AMSC Candidacy Presentation

21. TSF Properties • What is TSF? • TSF – Trilinos Solver Framework • High level matrix/block matrix manipulation language • Provides framework for integrating different solvers/preconditioners • Interface for representation-independent solvers • Why TSF? • Abstract interfaces for vectors and operators • Composable Block operators • Deferred inverse and transpose • Overloaded operators (matlab like syntax) • Matlab-like simplicity, running on a supercomputer • Transparent memory management AMSC Candidacy Presentation

22. Benchmark Problems • Lid Driven Cavity • Contains many features of harder flows • Steady and Unsteady Solutions • Flow over a diamond obstruction • Inflow/Outflow boundary conditions • Harder flow • MPSalsa • Realistic massively parallel, chemically reactive fluid flow code AMSC Candidacy Presentation

23. MPSalsa Steady Problem Results 2D Lid driven cavity on a 64 x 64 grid The values in each column represent the average number of Outer Saddle Point Solves per Newton Step. Residual reduction for each of the preconditioners. AMSC Candidacy Presentation

24. MPSalsa Steady Problem Results 2D Lid driven cavity Preliminary Time Comparison Mesh Independence AMSC Candidacy Presentation

25. Implementation Challenge • Timings are not very good • After profiling, upwards of 50% of the CPU time is spent in an inefficient memory allocation routine • A multigrid smoother is inefficiently implemented AMSC Candidacy Presentation

26. MPSalsa Steady Problem Results 2D Flow over a Diamond Obstruction The values in each column represent the average number of Outer Saddle Point Solves per Newton Step. AMSC Candidacy Presentation

27. Future Work • Sparse Approximate Commutator (SPAC) for Fp • Compare/Optimize CPU Time amongst methods • Tests on higher Re # for both steady/time dependent problems • More realistic problems • 3D Problems • Chemically reacting flow • Turbulent flows AMSC Candidacy Presentation

28. Conclusions • Incompressible Navier-Stokes Equations • Preconditioning the N-S Equations • General Preconditioners • Problem Specific • Pressure Correction Methods • Pressure Convection-Diffusion • Preliminary Remarks • ILU preconditioner does not scale well • Fp preconditioner is mesh independent and competitive in CPU time AMSC Candidacy Presentation

29. References • A.J. Chorin, A numerical method for solving incompressible viscous problems, Journal of Computational Physics, 2:12,1967. • H. C. Elman, D. J. Silvester and A. J. Wathen, Finite Elements and Fast Iterative Solvers, Oxford University Press, 2005. Howard Elman, V. E. Howle, John Shadid and Ray Tuminaro, A Parallel Block Multi-level Preconditioner for the 3D Incompressible Navier-Stokes Equations. Journal of Computational Physics 187:504-523, 2003. • D. Kay, D. Loghin, and A. J. Wathen, 2002, A preconditioner for the steady-state Navier-Stokes equations. SIAM J. Sci. Comput. 24, pp. 237-256. • M. Pernice and M.D. Tocci, A multigrid-preconditioned Newton Krylov method for the incompressible Navier-Stokes equations., SIAM J. Sci. Comput. 123, pp. 398-418. • S.V. Patankar, Numerical heat transfer and fluid flow, Hemisphere Pub. Corp, New York, 1980. AMSC Candidacy Presentation