ML: A Multilevel Preconditioning Package

1. ML: A Multilevel Preconditioning Package Copper Mountain Conference on Iterative Methods March 29-April 2, 2004 Jonathan Hu Ray Tuminaro Marzio Sala Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,for the United States Department of Energy under contract DE-AC04-94AL85000.

2. Outline • Overview • Multigrid basics • Available user options • Configuring & building • Interoperability with other packages • Example program • Conclusions

3. ML Package • Provides parallelmultigrid preconditioning for linear solver methods • Current developers: Ray Tuminaro, Jonathan Hu, Marzio Sala • Former: Charles Tong (LLNL) • Main methods • Geometric • Grid refinement hierarchy • 2-level FE basis function domain decomposition • AMG (algebraic multigrid based on aggregation) • Smoothed aggregation* • Edge-element AMG for Maxwell’s equations* • n-level (smoothed) aggregation domain decomposition * • Classical AMG • Written primarily in C • C++ interfaces to various Trilinos packages • 102,411 lines of code (as of this afternoon) • 26 example programs • Downloadable as part of Trilinos • CVS, bugzilla

4. W Cycle FMV Cycle Recursive MG Algorithm (V Cycle) V Cycle to solve A4u4=f4 MG(f, u, k) { if (k == 1) u1 = (A1)-1f1 else { Sk(Ak, fk,uk) //pre-smooth rk = fk – Ak uk fk-1 = Rk-1 rk; uk-1 = 0 //restrict MG(fk-1, uk-1, k-1) uk = uk + Ik-1 uk-1 //interpolate & correct Sk(Ak, fk,uk) //post-smooth } } Ak, Rk, Ik, Sk required on all levels Sk: smoothers Rk: restriction operators Ik: interpolation operators k=4 k=1

5. ML Capabilities • MG cycling: V, W, full V, full W • Grid Transfers • Several automatic coarse grid generators • Several automatic grid transfer operators • Coarse “grid” visualization capabilities • Smoothers • Jacobi, Gauss-Seidel, Hiptmair, Krylov methods, sparse approximate inverses, Chebyshev • Variety of Serial & Parallel Direct Solvers • SuperLU, Umfpack, KLU, MUMPS, etc. • Kernels: matrix/matrix multiply, etc.

6. Smoothed Aggregation • Developed for linear elasticity (Vanek, Mandel, Brezina) • Construct tentative prolongator tP • Aggregation: group unknowns together • Augment: interpolate null space (e.g., rigid body modes) • Construct final prolongator P • Smooth: P = (I –  D-1A)TP • lower energy in basis functions

7. 5 5 2 2 7 2 7 Take null space (e.g., const.) 5 7 2 Split into local basis functions Smooth Smoothed Aggregation

8. Uncoupled / MIS Aggregation • Greedy algorithm

9. Graph partitioning aggregation • Global graph partitioning • Operates on global domain • ParMETIS • Aggregates can span processors • Local graph partitioning • Processors work independently • METIS •  1 aggregate per processor

10. ML Smoother Choices • Jacobi, Point/Block Gauss Seidel • MLS • Based on Chebyshev polynomials of smoothed operator. • Serial: competitive with true Gauss-Seidel • Parallel: Performance is independent of # processors • Hiptmair • Distributed relaxation smoother for Maxwell’s Eqns. • Other smoothers used within each projection step • Aztec solvers • Krylov methods, incomplete factorizations • Direct solution (via Amesos interface): • UMFPACK, KLU (serial) • SuperLU • MUMPS

11. Via Epetra & TSF Accepts user data as Epetra objects Aztecoo TSF Can be wrapped as Epetra_Operator ML Meros TSF interface exists Epetra Amesos interface for direct solvers Amesos Other solvers Accepts other solvers and MatVecs Other matvecs ML and Other Packages

12. Configuring and Building ML • Builds by default when you configure & build Trilinos • By default, you get • Epetra & Aztecoo support • Example suite (Trilinos/packages/ml/examples) • Some options of interest (off by default) • MPI support • Graph partitioning aggregation (METIS, ParMETIS) • Direct solvers (Amesos) • Profiling • configure --with-mpi-compilers=/usr/local/mpich/bin • --with-ml_amesos \ • --with-libs=“-lamesos –lsuperlu”

13. A Small Example:ml/examples/ml_example_epetra_preconditioner.cpp Solve Ax=b: A: advection/diffusion operator Linear solver: gmres Precond.: 2-level AMG, graph-partitioning aggregation Solution component Example methods Packages used Linear Solver AztecOO (Epetra) gmres ML: multi- grid pre- cond. ML (Teuchos) AMG

14. ml_example_epetra_preconditioner.cpp Trilinos_Util_CrsMatrixGallery Gallery(“recirc_2d", Comm); Gallery.Set("problem_size", 10000); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); // Set some solver options solver.SetAztecOption(AZ_solver, AZ_gmres); solver.SetAztecOption(AZ_output, 10); solver.SetAztecOption(AZ_kspace, 160);

15. example (contd.) // Set up multilevel preconditioner ParameterList MLList;// parameter list for ML options MLList.set("max levels",2); MLList.set("aggregation: type", "METIS"); // graph partitioning MLList.set("aggregation: nodes per aggregate", 16); // set up aztecoo smoother MLList.set("smoother: type","aztec"); int options[AZ_OPTIONS_SIZE]; double params[AZ_PARAMS_SIZE]; AZ_defaults(options,params); options[AZ_precond] = AZ_dom_decomp; options[AZ_subdomain_solve] = AZ_ilut; MLList.set("smoother: aztec options", options); MLList.set("smoother: aztec params", params); MLList.set("coarse: type","Amesos_Superludist"); MLList.set("coarse: max processes", 4); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times

16. Other problem types: “DD”, “DD-ML”, “maxwell” AMG for Common Problem Types Trilinos_Util_CrsMatrixGallery Gallery(“laplace_3d", Comm); Gallery.Set("problem_size", 100*100*100); // linear system matrix & linear problem Epetra_RowMatrix * A = Gallery.GetMatrix(); Epetra_LinearProblem * Problem = Gallery.GetLinearProblem(); // Construct outer solver object AztecOO solver(*Problem); solver.SetAztecOption(AZ_solver, AZ_cg); // Set up multilevel precond. with smoothed aggr. defaults ParameterList MLList; // parameter list for ML options ML_Epetra::SetDefaults(“SA”,MLList); ML_Epetra::MultiLevelPreconditioner * MLPrec = new // create preconditioner ML_Epetra::MultiLevelPreconditioner(*A, MLList, true); solver.SetPrecOperator(MLPrec); // tell solver to use ML preconditioner solver.Iterate(500, 1e-12); // iterate at most 500 times

17. Collaborations • Within Sandia • ALEGRA • Radiation • Maxwell • Electrostatic potential • MPSalsa (Shadid, et al.) • CEPTRE • External users • EM3D (Lawrence Berkeley NL) • P. Arbenz (ETH Zurich) • R. Geus (PSI) • J. Fish, H. Waisman (RPI) • Potential Users • PREMO (Sandia)

18. Getting Help • Website: www.cs.sandia.gov/~tuminaro/ml • See Trilinos/packages/ml/examples • See guide: • ML User’s Guide, ver. 3.0 (in ml/doc) • Mailing lists • ml-users@software.sandia.gov • ml-announce@software.sandia.gov • Bug reporting, enhancement requests via bugzilla: • http://software.sandia.gov/bugzilla • Email us directly • jhu@sandia.gov • rstumin@sandia.gov • msala@sandia.gov

19. Conclusions • ML 3.0 release this May • Part of next Trilinos release • Existing options • Uncoupled / MIS aggregation • Smoothers • SuperLU • New options with ML 3.0 • Graph-based aggregation • “parameter list” setup • Amesos interface to direct solvers • Visualization, aggregate statistics • Use ML with Trilinos! • ML as preconditioner is trivial • Rich set of external libraries