Bill Spotz DOE/Office of Science/ASCR (Sandia National Laboratories) SIAM Annual Meeting 2008

1. PyTrilinos: A Python Interface to Parallel, Object-Oriented Solver Packages for Scientific Computing Bill Spotz DOE/Office of Science/ASCR (Sandia National Laboratories) SIAM Annual Meeting 2008 July 10, 2008

2. Chemically reacting flows Climate modeling Combustion Compressible flows Computational biology Electrical modeling Heat transfer Load balancing Materials modeling MEMS modeling Mesh generation Optimization and uncertainty quantification Seismic imaging Shock and multiphysics Structural dynamics Computational Sciences at Sandia

3. The Trilinos Project • Provide a central repository for Sandia’s solver technology • Increase code-reuse • Organized on concept of “packages” • Minimize package interdependence • Maximize package interoperability • Provide a framework for SQE and SQA • Compliance with requirements • Nightly test harness • High degree of developer autonomy • Open source: GNU Lesser General Public License • Web site: http://trilinos.sandia.gov • Next release: Version 9.0, September, 2008 • Trilinos Users Group Meeting every November

4. Communicators, maps, vectors, matrices, graphs… Direct, sparse Complex interface to AztecOO Linear Algebra Services Java Sparse kernels Iterative, sparse Tools & Utilities Templated Reduction, transformation operators Direct, dense Iterative, sparse, next generation Extensions Generic interface Linear Solvers Nonlinear Solvers Preconditioners Meta-Solvers Primary tools package Python Interface Common testing tools Web interface Matrix gallery Incomplete factorization Eigensolvers Common interface Multi-level Time-stepping algorithms Partitioning & load balance Domain decomposition Continuation algorithms Mortar elements Package template Segregated, block Constrained optimization Trilinos examples The Trilinos Project Epetra Jpetra Tpetra EpetraExt Kokkos RTOp Teuchos TriUtils Galeri Thyra Isorropia Moertel Didasko PyTrilinos WebTrilinos New_Package Amesos AztecOO Belos Komplex Pliris Stratimikos NOX Anasazi Rythmos LOCA MOOCHO IFPACK ML Claps Meros

5. The Trilinos Project • Recently, the scope of Trilinos has been expanded to include: • Meshing • Numerical methods • Finite element interface • Dynamic load balancing • Automatic differentiation • Probably others…

6. PyTrilinos • Linear Algebra Services • Epetra (with extensive NumPy compatibility and integration) • EpetraExt (coloring algorithms and some I/O) • Linear Solvers • Amesos (LAPACK, KLU, UMFPACK, ScaLAPACK, SuperLU, SuperLUDist, DSCPACK, MUMPS) • AztecOO • Preconditioners • IFPACK • ML • Nonlinear Solvers • NOX • Meta-Solvers • LOCA (python wrappers not yet caught up to recent redesigns) • Anasazi • Tools and Utilities • Teuchos (ParameterLists, communicators, traits and refcount pointers) • TriUtils • Galeri • Thyra (early development stage)

7. PyTrilinos Design • PyTrilinos is a “python package,” meaning a python module with sub-modules • Each “Trilinos package” translates into a PyTrilinos sub-module • Trilinos package namespaces also become sub-modules • PyTrilinos python interfaces match Trilinos C++ interfaces (within reason) • Generated by SWIG (simple wrapper interface generator) • Some classes made more “pythonic”

8. PyTrilinos Documentation • Trilinos documentation is handled by doxygen: special comments within code • Web pages updated twice daily • C++ doxygen comments are now automatically converted to python doc strings: >>> from PyTrilinos import Epetra >>> help(Epetra.Vector) Returns standard Epetra_Vector documentation

9. Communicators Comm SerialComm MpiComm PyComm Maps BlockMap Map LocalMap Vectors MultiVector Vector IntVector SerialDense objects SerialDenseOperator SerialDenseMatrix SerialDenseVector SerialDenseSolver IntSerialDenseMatrix IntSerialDenseVector Graphs CrsGraph Operators Operator RowMatrix CrsMatrix PyTrilinos.Epetra

10. A Quick Detour… • Python lists are not suitable for scientific computing • Flexible but inefficient • Heterogeneous data, noncontiguous memory • NumPy module provides needed functionality • Contiguous, homogeneous n-dimensional arrays • High-level interface • Part of SciPy • SciPy is a large, open source package for a wide variety of python interfaces to scientific software: • NetLib’s “greatest hits”

11. PyTrilinos.Epetra and NumPy • Array-like classes inherit from numpy.UserArray • MultiVector • Vector • IntVector • SerialDenseMatrix • SerialDenseVector • IntSerialDenseMatrix • IntSerialDenseVector • Methods throughout Epetra have arguments that accept or produce pointers to C arrays • Python input arguments accept python sequences • Python output arguments produce ndarrays

12. PyTrilinos.Teuchos • Teuchos::ParameterList • Used by several Trilinos packages to set problem parameters • Maps string names to arbitrary-type values • Python implementation allows dictionary substitutions • The following conversions are supported:

13. PyTrilinos Demonstration • Governing equation: • Boundary conditions: • Exact solution: • CDS: • Oscillations:

14. Parallelism • Configure Trilinos with –enable-mpi, and PyTrilinos will also be mpi-enabled • Scripts are run in parallel in the standard way: $ mpirun -np 4 python script.py … • When Epetra is imported, MPI_Init() is called automatically and MPI_Finialize() is registered with the atexit module. • Use the PyComm() function to obtain a communicator, and use a parallel programming paradigm: • Global and local IDs • Ghost nodes if necessary • Epetra Import and Export objects if needed

15. Conclusions • Python lets developers focus on the problem • Memory management, garbage collection • Powerful, flexible containers • Clean, readable syntax • PyTrilinos provides access to powerful solver technologies • Rapid prototyping • Application development • Serial, parallel