Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl

1. The U.S. DOE Advanced CompuTational Software (ACTS) Collection Tony Drummond Lawrence Berkeley National Laboratory LADrummond@lbl.gov SIMULA Research Laboratory - May 2008

2. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

3. http://acts.nersc.gov/MatApps Where are the applications? Development of High End Computer Simulations • Accelerator Science • Astrophysics • Biology • Chemistry • Earth Sciences • Materials Science • Nanoscience • Plasma Science • : • Commonalities: • Major advancements in Science • Increasing demands for computational power • Rely on available computational systems, • languages, and software tools SIMULA Research Laboratory - May 2008

4. min[time_to_solution] (production) (Software Evolution) (Long-term deliverables) min[software-development-cost] max[software_life] and max[resource_utilization] Software Development and Evolution min[time_to_first_solution] (prototype) • Outlive Complexity • Increasingly sophisticated models • Model coupling • Interdisciplinary • Sustained Performance • Increasingly complex algorithms • Increasingly diverse architectures • Increasingly demanding applications SIMULA Research Laboratory - May 2008

5. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

6. THE U.S. DOE ACTS COLLECTION Goal: The Advanced CompuTational Software Collection (ACTS) makes reliable and efficient software tools more widely used, and more effective in solving the nation’s engineering and scientific problems. • References: • L.A. Drummond, O. Marques: An Overview of the Advanced CompuTational Software (ACTS) Collection. ACM Transactions on Mathematical Software Vol. 31 pp. 282-301, 2005 • http://acts.nersc.gov SIMULA Research Laboratory - May 2008

7. The Advanced CompuTational Software Collection (ACTS) • Components: • Solid Base: non-commercial and open source tools developed at DOE laboratories and universities. • Independent Tool Evaluations and Consultation provided through acts-support@nersc.gov • High Level User Support problem identification, tool and interface selection, specific tuning parameter configurations, installation, documentation, etc. • Training and Dissemination workshops, lectures, active conference participation (acts.nersc.gov. • Collaborations with HPC centers, computational sciences research centers (national and international level), and software and computer vendors. SIMULA Research Laboratory - May 2008

8. SIMULA Research Laboratory - May 2008

9. Changes in algorithms sometimes lead to several years advancement in computations. Needs Flexibility! Its performance is influenced by system parameters and in steps in the algorithm. Critical points: portability and scalability. New Architecture requires extensive tuning, may even require new programming paradigms. This is Difficult to maintain and not “very” portable. Software Sustainability Algorithmic Implementations I/O Application Data Layout Control Tuned and machine Dependent modules SIMULA Research Laboratory - May 2008

10. Software Sustainability USER's APPLICATION CODE (Main Control) Compilers + Expert Drivers + Support AVAILABLE AVAILABLE Algorithmic Implementations AVAILABLE I/O Application Data Layout LIBRARIES LIBRARIES & PACKAGES LIBRARIES & PACKAGES Tuned and machine Dependent modules SIMULA Research Laboratory - May 2008

11. Scientific or engineering context • Domain expertise Critical Path for HPC Software Stack • Simulation codes • Data Analysis codes General Purpose Libraries • Algorithms • Data Structures • Code Optimization • Programming Languages • O/S - Compilers Hardware - Middleware - Firmware SIMULA Research Laboratory - May 2008

12. Funded by DOE/ASCR Library Development Numerical Tools Code Development Run Time Support http://acts.nersc.gov • Algorithms • Data Structures • Code Optimization • Programming Languages • O/S - Compilers Critical Path for HPC Software Stack General Purpose Libraries Hardware - Middleware - Firmware SIMULA Research Laboratory - May 2008

13. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

14. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

15. Structure of PETSc PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK SIMULA Research Laboratory - May 2008

16. Linear System Interfaces Linear Solvers GMG, ... FAC, ... Hybrid, ... AMGe, ... ILU, ... Data Layout structured composite block-struc unstruc CSR Hypre Conceptual Interfaces SIMULA Research Laboratory - May 2008

17. Hypre Conceptual Interfaces to Solvers List of Solvers and Preconditioners per Conceptual Interface SIMULA Research Laboratory - May 2008

18. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

19. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

20. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

21. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

22. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

23. TAO - Interface with PETSc SIMULA Research Laboratory - May 2008

24. OPT++ Interfaces • Four major classes of problems available • NLF0(ndim, fcn, init_fcn, constraint) • Basic nonlinear function, no derivative information available • NLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information available • FDNLF1(ndim, fcn, init_fcn, constraint) • Nonlinear function, first derivative information approximated • NLF2(ndim, fcn, init_fcn, constraint) • Nonlinear function, first and second derivative information available SIMULA Research Laboratory - May 2008

25. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

26. ACTS Numerical Tools: Functionality SIMULA Research Laboratory - May 2008

27. ACTS Tools: Functionality SIMULA Research Laboratory - May 2008

28. ACTS Tools: Functionality SIMULA Research Laboratory - May 2008

29. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

30. How Does One Use ACTS Tools? Linear System Interfaces • -ksp_type [cg,gmres,bcgs,tfqmr,…] • -pc_type [lu,ilu,jacobi,sor,asm,…] More advanced: • -ksp_max_it <max_iters> • -ksp_gmres_restart <restart> • -pc_asm_overlap <overlap> • -pc_asm_type <. . > Linear Solvers GMG FAC Hybrid, ... AMGe ILU, ... Data Layout structured composite blockstrc unstruc CSR CALL BLACS_GET( -1, 0, ICTXT ) CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL ) : CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL ) : : CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB, $ INFO ) Language Calls Command lines Problem Domain SIMULA Research Laboratory - May 2008

31. TAU PETSc Ex 2 Ex 1 Tool to Tool Interoperability One Side Interoperability TOOL B TOOL C TOOL A TOOL F TOOL E TOOL D SIMULA Research Laboratory - May 2008

32. PyACTS matlabMPI Star-P NetSolve High Level Interfaces High-level User Interfaces to the ACTS Collection Ax = b View_field(T1) User OPT++ PAWS CUMULVS TAU Globus Hypre Global Arrays Chombo AZTEC PETSc PVODE Overture ScaLAPACK SuperLU TAO SIMULA Research Laboratory - May 2008

33. PyACTS Vicente Galiano Miguel Hernandez University Tony Drummond Lawrence Berkeley National Laboratory Violeta Migallón and José Penadés University of Alicante Goal: Provide a didactical tool to the ACTS collection. Provide a Python based interface to the ACTS Collection. • References: • L. A. Drummond, V. Galiano, O. Marques, V. Migallon, J.Penades: PyACTS: A High-level Framework for Fast Development of High Performance Applications. Lecture Notes in Computer Sciences, Vol. 4395, pp 417-425, 2007. SIMULA Research Laboratory - May 2008

34. PyACTS PyACTS PyScaLAPACK PySuperLU PyACTS Wrappers ScaLAPACK Wrappers SuperLU Wrappers Python World PyMPI NumPy . . . ScaLAPACK SuperLU Python SIMULA Research Laboratory - May 2008

35. PyACTS: Basic Services • BASIC Services: Creation and modification of different data objects and parallel environment specifications (matrices, data layouts, ctx,) • I/O Services : Parallel read/write. Currently supported ASCII and NetCDF. • Verification and Validation: Predicates and parameter type checking. • Data Conversion. Interoperable objects between libraries. SIMULA Research Laboratory - May 2008

36. PyACTS: Motivation PyClimate (J. Saenz et al,Univ. Basque Country) • Support to common tasks during the analysis of climate variability data. • Simple IO operations • Operations with COARDS-compliant NetCDF files • Empirical Orthogonal Function (EOF) analysis, • Canonical Correlation Analysis (CCA) • Singular Value Decomposition (SVD) analysis of coupled datasets • Some linear digital filters • Kernel based probability-density function estimation and • access to DCDFLIB.C library from Python. SIMULA Research Laboratory - May 2008

37. PyACTS: Performance in PyClimate EOF calculations Empirical Orthogonal Function (Day calc) SIMULA Research Laboratory - May 2008

38. PyScaLAPACK: pvgesvd Performance SIMULA Research Laboratory - May 2008

39. PyACTS: Performance > from PyACTS import * > import PyACTS.PyPBLAS as PyPBLAS > import time > n=500 > ACTS_lib=1 # ScaLAPACK library > PyACTS.gridinit() # grid initialization > alpha=Scal2PyACTS(2,ACTS_lib) # convert scalar # to PyACTS scalar > beta=Scal2PyACTS(3,ACTS_lib) > a=Rand2PyACTS(n,n,ACTS_lib) # generate a random # PyACTS array > b=Rand2PyACTS(n,n,ACTS_lib) > c=Rand2PyACTS(n,n,ACTS_lib) > c=PyPBLAS.pvgemm(alpha,a,b,beta,c) # call level 3 # PBLAS routine > PyACTS.gridexit() c=PyPBLAS.pvgemm(alpha,a,b,beta,c) SIMULA Research Laboratory - May 2008

40. OUTLINE • Motivation • Introduction to the DOE ACTS Collection • Interfaces to the ACTS Collection • Software Sustainability Requirements • References SIMULA Research Laboratory - May 2008

41. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

42. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

43. Problem Statement: Software Sustainability • THE GOOD • Many successful HPC stories have induced major advances in science and engineering • We have successful run and scale applications on 100000+ processors • THE BAD • Portability Across Platforms is Still An Outstanding Issue: • Readiness • Performance • Robustness and Correctness • THE UGLY • Multi-Core and Many Core Era is knocking at the HPC door SIMULA Research Laboratory - May 2008

44. Versions (tools, systems, O/S, compilers) • Sanity-check (robustness) • Interoperability (maintained) • Consistent Documentation Software Quality Assurance • Robustness • Scalability • Extensibility • Interoperability • User Friendliness • Documentation • Periodic test and evaluations • (test engines and dependency graphs) SIMULA Research Laboratory - May 2008

45. ScaLAPACK PBLAS Global Local LAPACK BLACS platform specific BLAS MPI/PVM/... ScaLAPACK’s Software Structure SIMULA Research Laboratory - May 2008

46. + * 2.2 GHz AMD Opteron * * + BLAS: Basic Linear Algebra Subroutines • Level 1 BLAS: vector-vector • Level 2 BLAS: matrix-vector • Level 3 BLAS: matrix-matrix BLAS LEVELS: • Design Considerations: • Portability • Performance: development of blocked algorithms is important for performance! SIMULA Research Laboratory - May 2008

47. ScaLAPACK: Data Layouts • 1D block and column distributions • 1D block-cycle column and 2D block-cyclic distribution • 2D block-cyclic distribution used in ScaLAPACK for dense matrices SIMULA Research Laboratory - May 2008

48. Astrophysics Applications Cosmic Microwave Background Analysis, BOOMERanG collaboration, MADCAP code (Apr. 27, 2000). • The statistics of the tiny variations in the CMB (the faint echo of the Big Bang) allows the determination of the fundamental parameters of cosmology to the percent level or better. • MADCAP (Microwave Anisotropy Dataset Computational Analysis Package) • Makes maps from observations of the CMB and then calculates their angular power spectra. (See http://crd.lbl.gov/~borrill). • Calculations are dominated by the solution of linear systems of the form M=A-1B for dense nxn matrices A and B scaling as O(n3) in flops. MADCAP uses ScaLAPACK for those calculations. SIMULA Research Laboratory - May 2008

49. PETSc PDE Application Codes ODE Integrators Visualization Nonlinear Solvers, Unconstrained Minimization Interface Linear Solvers Preconditioners + Krylov Methods Object-Oriented Matrices, Vectors, Indices Grid Management Profiling Interface Computation and Communication Kernels MPI, MPI-IO, BLAS, LAPACK PETSc Image Provided by PETSc Development Team, ANL) SIMULA Research Laboratory - May 2008

50. Synchronization Points Basic Conjugate Gradient Algorithm Scalars , , y  Vectors x, r, p (= search direction), and q SIMULA Research Laboratory - May 2008