Introduction to the ACTS Collection Why/How Do We Use These Tools?

1. Introduction to the ACTS Collection Why/How Do We Use These Tools? Tony Drummond Computational Research Division Lawrence Berkeley National Laboratory University of California, Berkeley - CS267 April 20, 2006 - Berkeley, California

2. Lesson = High QualitySoftware Reusability Funded by DOE/ASCR Library Development Numerical Tools Code Development Run Time Support http://acts.nersc.gov General Purpose Libraries • Algorithms • Data Structures • Code Optimization • Programming Languages • O/S - Compilers Introduction to The ACTS Collection Hardware - Middleware - Firmware The DOE ACTS Collection (http://acts.nersc.gov)

3. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

4. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

5. Structure of PETSc PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK The DOE ACTS Collection (http://acts.nersc.gov)

6. Linear System Interfaces Linear Solvers GMG, ... FAC, ... Hybrid, ... AMGe, ... ILU, ... Data Layout structured composite block-struc unstruc CSR Hypre Conceptual Interfaces The DOE ACTS Collection (http://acts.nersc.gov)

7. Hypre Conceptual Interfaces INTERFACE TO SOLVERS List of Solvers and Preconditioners per Conceptual Interface The DOE ACTS Collection (http://acts.nersc.gov)

8. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

9. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

10. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

11. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

12. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

13. TAO - Interface with PETSc The DOE ACTS Collection (http://acts.nersc.gov)

14. OPT++ Interfaces • Four major classes of problems available • NLF0(ndim, fcn, init_fcn, constraint) • Basic nonlinear function, no derivative information available • NLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information available • FDNLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information approximated • NLF2(ndim, fcn, init_fcn, constraint) • Nonlinear function, first and second derivative information available The DOE ACTS Collection (http://acts.nersc.gov)

15. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

16. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

17. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

18. ACTS Numerical Tools: Functionality The DOE ACTS Collection (http://acts.nersc.gov)

19. CALL BLACS_GET( -1, 0, ICTXT ) CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL ) : CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL ) : : CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB, $ INFO ) Library Calls Linear System Interfaces • -ksp_type [cg,gmres,bcgs,tfqmr,…] • -pc_type [lu,ilu,jacobi,sor,asm,…] More advanced: • -ksp_max_it <max_iters> • -ksp_gmres_restart <restart> • -pc_asm_overlap <overlap> Linear Solvers GMG FAC Hybrid, ... AMGe ILU, ... Problem Domain Data Layout structured composite blockstrc unstruc CSR Introduction to The ACTS Collection User Interfaces Command lines The DOE ACTS Collection (http://acts.nersc.gov)

20. High Level Interfaces OPT++ PAWS CUMULVS TAU Globus Hypre Global Arrays Chombo AZTEC PETSc PVODE Overture ScaLAPACK SuperLU TAO Introduction to The ACTS Collection User Interfaces PyACTS Ax = b matlab*P Star-P View_field(T1) User NetSolve The DOE ACTS Collection (http://acts.nersc.gov)

21. TAU PETSc Ex 2 Ex 1 Tool Interoperability Tool-to-Tool TOOL B TOOL C TOOL A TOOL F TOOL E TOOL D Trilinos Ex 3 The DOE ACTS Collection (http://acts.nersc.gov)

22. Component Technology! Tool C Tool A Tool D Tool B ESI CCA The DOE ACTS Collection (http://acts.nersc.gov)

23. Introduction to PETSc • Introducción a PETSC • Vectors • Matrices • PETSc viewers • Use of Preconditioners in PETSc • Some examples using PETSc The DOE ACTS Collection (http://acts.nersc.gov)

24. What is PETSc? What is PETSc? PETSc: Portable Extensible Toolkit for Scientific Computation • Developed at Argonne National Laboratory • Library that supports work with PDE • Public Domain • Portable to a variety of platforms • Project started in 1991. • Written in C with a fortran interface and OO flavor The DOE ACTS Collection (http://acts.nersc.gov)

25. PETSc Structure PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK The DOE ACTS Collection (http://acts.nersc.gov)

26. Some Algorithmic Implementations in PETSc Nonlinear Solvers Time Steppers Newton-based Methods Other Euler Backward Euler Pseudo Time Stepping Other Line Search Trust Region Krylov Subspace Methods Richardson Chebychev Other GMRES CG CGS Bi-CG-STAB TFQMR Preconditioners Additive Schwartz Block Jacobi Jacobi ILU ICC LU (Sequential only) Others Matrices Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) Dense Matrix-free Other Distributed Arrays Index Sets Indices Block Indices Stride Other Vectors The DOE ACTS Collection (http://acts.nersc.gov)

27. Vectors and Matrices in PETSc Vectors Fundamental Objects to store fields, right-hand side vectors, etc. . • Matrices • Fundamental Objects to store operators (like Jacobians) The DOE ACTS Collection (http://acts.nersc.gov)

28. Basic Vector Operations proc 0 proc 1 proc 2 proc 3 proc 4 • Each process owns a piece of the vector • VECTOR Types: Sequential, MPI or SHARED • VecCreate(MPI_Comm Comm,Vec * v) • comm - MPI_Comm parallel processes • v = vector • VecSetType(Vec,VecType) • Valid Types are: • VEC_SEQ, VEC_MPI, o VEC_SHARED • VecSetSizes(Vec *v,int n, int N) • Where n or N (not both) can be PETSC_DECIDE • VecDestroy(Vec *) The DOE ACTS Collection (http://acts.nersc.gov)

29. Basic Vector Operations VecCreateMPI(PETSC_COMM_WORLD, m, n, x); PETSc - Vector Example (in C) • #include petscvec.h • int main(int argc,char **argv) • { • Vec x; • int n = 20,m=4, ierr; • PetscInitialize(&argc,&argv); • VecCreate(PETSC_COMM_WORLD,&x); • VecSetSizes(x,PETSC_DECIDE,n); • VecSetFromOptions(x); • <-- USER_CODE:Work with PETSc VECTOR --> • PetscFinalize(); • return 0; • } The DOE ACTS Collection (http://acts.nersc.gov)

30. Basic Operations with Vectors Function Name Operation VecAXPY(Scalar *a, Vec x, Vec y) y = y + a*x y = x + a*y VecAYPX(Scalar *a, Vec x, Vec y) w = a*x + y VecWAXPY(Scalar *a, Vec x, Vec y, Vec w) x = a*x VecScale(Scalar *a, Vec x) y = x VecCopy(Vec x, Vec y) w_i = x_i *y_i VecPointwiseMult(V ec x, Vec y, Vec w) r = max x_i VecMax(Vec x, int *idx, double *r) x_i = s+x_i VecShift(Scalar *s, Vec x) x_i = |x_i | VecAbs(Vec x) r = ||x|| VecNorm(Vec x, NormType type , double *r) The DOE ACTS Collection (http://acts.nersc.gov)

31. Basic Matrix Operations Types: • default sparse AIJ: MPIAIJ, SEQAIJ • block sparse AIJ (for multi-component PDEs): MPIAIJ, SEQAIJ • symmetric block sparse AIJ: MPISBAIJ, SAEQSBAIJ • block diagonal: MPIBDIAG, SEQBDIAG • dense: MPIDENSE, SEQDENSE • matrix-free • MatCreate(MPI_Comm Comm, Mat *A) • MPI_Comm - group of process that work on A • MatSetSizes(Mat A, PetscInt m, PetscInt n, PetscInt M, PetscInt N) • MatSetType(Mat,MatType) • MatDestroy(Mat) The DOE ACTS Collection (http://acts.nersc.gov)

32. Basic Matrix Operations • Single user interface to matrices (operators) • Input values to a PETSc matrix • MatSetValues(Mat mat,PetscInt m,const PetscInt idxm[],PetscInt n,const PetscInt idxn[],const PetscScalar v[],InsertMode addv) • Matriz-vector multiplication For instance: MatMult(A,y,x) (y≠x) • Matrix viewing • MatView() • Multiple underlying implementations • AIJ, block AIJ, symmetric block AIJ, block diagonal, dense, matrix-free, etc. The DOE ACTS Collection (http://acts.nersc.gov)

33. Basic Matrix Operations • Block-Row distribution per process M=8,N=8,m=3,n=k1 rstart=0,rend=4 proc 1 M=8,N=8,m=3,n=k2 rstart=3,rend=6 proc 2 M=8,N=8,m=2,n= k3 rstart=6,rend=8 proc 3 The values of k1, k2 y k3 depend on the application and must be compatible with vector x (Ax=b) MatGetOwnershipRange(Mat A, int *rstart, int *rend) • rstart: first global row number that belongs to a process • rend -1: Last row global number that belongs to a process The DOE ACTS Collection (http://acts.nersc.gov)

34. PETSc Matrix Example • Included in PETSc Distribution: • $PETSC_DIR/src/mat/tutorial/ex2.c • $PETSC_DIR/src/mat/tutorial/ex5.c The DOE ACTS Collection (http://acts.nersc.gov)

35. Basic Viewer Operations • Obtain basic information on PETSc viewers • Run time options • Extract data that can be later use by other packages/app. programs. • vector fields, matrix contents • various formats (ASCII, binary) • Visualization • simple graphics created with X11 to display vector campo, non-zero element structure of a matrix, etc. The DOE ACTS Collection (http://acts.nersc.gov)

36. Basic Viewer Operations • MatView(Mat A, PetscViewer v); • Runtime options available after matrix assembly -mat_view_info Information related to matrix type -mat_view_draw nonzero pattern -mat_view data in ASCII etc. The DOE ACTS Collection (http://acts.nersc.gov)

37. Examples of Viewers and Matrices • Included in PETSc Distribution: • $PETSC_DIR/src/mat/tests/ex2.c • Using the -mat_view_info_detailed, etc • $PETSC_DIR/src/mat/tests/ex3.c • Using the -mat-view-draw The DOE ACTS Collection (http://acts.nersc.gov)

38. PETSc KSP Objective: Support the solution of Linear System of the form, Ax=b, Krylov Subspace methods are highly dependent of the spectrum of the matrix A. The use of preconditioners can speed-up their convergence. KRYLOV SUBSPACE METHODS + PRECONDITIONERS R. Freund, G. H. Golub, and N. Nachtigal. Iterative Solution of Linear Systems,pp 57-100. ACTA Numerica. Cambridge University Press, 1992. The DOE ACTS Collection (http://acts.nersc.gov)

39. Right vs. Left Preconditioning For ML=I For MR=I PETSC Default Ax=b, The DOE ACTS Collection (http://acts.nersc.gov)

40. Using PETSc’s KSP Main Routine PETSc Linear Solvers (SLES) Solve Ax = b PC KSP Application Initialization Evaluation of A and b Post- Processing User code PETSc code The DOE ACTS Collection (http://acts.nersc.gov)

41. Conjugate Gradient GMRES CG-Squared Bi-CG-stab Transpose-free QMR Chebychev CGNE Many more (check manual) Block Jacobi Overlapping Additive Schwarz ICC, ILU via BlockSolve95 ILU(k), LU etc. Solvers and Preconditioners in PETSc Interface to Other Packages (like SuperLU, SuperLU_DIST, Hypre, etc) KSPPREONLY Krylov Methods (KSP) Preconditioners (PC) The DOE ACTS Collection (http://acts.nersc.gov)

42. Flow of Control for PDE Solution Main Routine Timestepping Solvers (TS) Nonlinear Solvers (SNES) Linear Solvers (KSP) PETSc PC Application Initialization Function Evaluation Jacobian Evaluation Post- Processing User code PETSc code The DOE ACTS Collection (http://acts.nersc.gov)

43. ODE Solvers (TS) in PETSc • Linear TS Example (ex3.c) • Non Linear TS Example (ex1.c) Time Stepper Types (TS) TS_EULER - Euler TS_SUNDIALS- SUNDIALS interface TS_BEULER - Backward Euler TS_PSEUDO- Pseudo-timestepping The DOE ACTS Collection (http://acts.nersc.gov)