Introduction Problem Description MPP Software Tools Computations Future Direction

1. Using the PETSc Parallel Software library in Developing MPP Software for Calculating Exact Cumulative Reaction Probabilities for Large Systems(M. Minkoff and A. Wagner)ANL (MCS/CHM) • Introduction • Problem Description • MPP Software Tools • Computations • Future Direction

2. Parallelization of Cumulative Reaction Probabilities (CRP) with PETScM. Minkoff (ANL/MCS) and A. Wagner(ANL/CHM) Calculation of gas phase rate constants • Develop a highly scalable and efficient parallel algorithm for calculating the Cumulative Reaction Probability, P. • Use parallel subroutine libraries for higher generation parallel machines to develop parallel CRP simulation software. • Implementing Miller and Manthe (1994) method for time-independent solution of P in parallel. • P is determined for an eigenvalue problem with an operator involving two Green’s functions. The eigenvalues are obtained using a Lanczos method. The Green’s functions are evaluated via a GMRES iteration with diagonal preconditioner.

3. Benefits of using PETSc • Sparsity: PETSc allows arbitrarily sparse data structures • GMRES: PETSc has GMRES as an option for linear solves • Present tests involve problems in dimensions 3 to 6. Testing is underway using an SGI Power Challenge (ANL), and SGI/CRAY T3E (NERSC). (Portability is provided via MPI and PETSc, so higher dimensional systems are planned for future work).

4. Chemical Dynamics Theory3 angles, 3 stretches6 degrees of freedom

5. Chemical Dynamics Theory • How fast do chemicals react? • Rate constant “k” determines it • d[X] / dt = -k1[X][Y] + k2[Z][Y] • many rates at work in devices • rates express interactions in the chemistry • individual rates are measurable and calculable • rates depend on T, P.

6. Chemical Dynamics TheoryN(E) = Tr[P(E)] • Rates are related to • Cumulative Reaction Probability (CRP),N(E) N(E) = 4 Tr[

7. Chemical Dynamics Theory • Probability Operator and It’s Inverse • Using probability method calculates a few large eigenvalues via iterative methods. The iterative evaluation involves the action of two Green’s function. • Using inverse probability method involves a direct calculation each iteration to obtain a few smallest eigenvalues. At each iteration the action of a vector by the Green’s function is required. This leads to solving linear systems involving the Hamiltonian.

8. Chemical Dynamics Theory • The Green’s functions have the form: G(E) = (E + ie - H)-1 and so we need to solve two linear systems (at each iteration) of the form: (E + ie - H)y = x where x is known. This system is solved via GMRES with preconditioning methods (initially diagonal scaling).

9. PETSc: Portable, Extensible Toolkit for Scientific Computing Satish Balay, William Gropp, Lois McInnes, and Barry Smith MCS Division, Argonne National Laboratory • Focus: data structures and routines for the scalable solution of PDE-based applications • Object-oriented design using mathematical abstractions • Freely available and supported research code • Available via http://www.mcs.anl.gov/petsc • Usable in C, C++, and Fortran77/90 (with minor limitations in Fortran 77/90 due to their syntax) • Users manual, hyperlinked manual pages for all routines • Many tutorial-style examples • Support via email: petsc-maint@mcs.anl.gov

10. PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK Application Codes Using PETSc Applications can interface to whatever abstraction level is most appropriate.

11. Nonlinear Solvers Time Steppers Newton-based Methods Other Euler Backward Euler Pseudo Time Stepping Other Line Search Trust Region Krylov Subspace Methods GMRES CG CGS Bi-CG-STAB TFQMR Richardson Chebychev Other Preconditioners Additive Schwartz Block Jacobi Jacobi ILU ICC LU (Sequential only) Others Matrices Compressed Sparse Row (AIJ) Blocked Compressed Sparse Row (BAIJ) Block Diagonal (BDIAG) Dense Other Vectors Index Sets Indices Block Indices Stride Other PETSc Numerical Components

12. Sample Scalable Performance 600 MHz T3E, 2.8M vertices • 3D incompressible Euler • Tetrahedral grid • Up to 11 million unknowns • Based on a legacy NASA code, FUN3d, developed by W. K. Anderson • Fully implicit steady-state • Newton-Krylov-Schwarz algorithm with pseudo-transient continuation • Results courtesy of Dinesh Kaushik and David Keyes, Old Dominion University

13. Computations via MPI and PETSc

14. 5D/T3E Results for Varying Eigenvalue and G-S Method

15. Parallel Speedup5D/6D ANL/SGI and NERSC/T3E

16. Storage Required for Higher Dimensions

17. Results and Future Work • Achieved parallelization with less effort • Suboptimal but perhaps 2X Optimal Performance • Testing for 6D and 7D underway. • MPP CPU and Memory can provide necessary resources • Many degrees of freedom can be approximated, so maximum dimension needed is ~10. • Develop block structured preconditioning methods