Manoj Krishnan Pacific Northwest National Laboratory

1. MCMD Programming and Implementation Issues Manoj Krishnan Pacific Northwest National Laboratory

2. Motivation • The challenges in developing large-scale applications are … • Addressing complexity • Improve productivity • Scaling to massive number of processors • How applications can exploit the massive amount of parallelism available in teraflop and petaflop-scale systems

3. Multilevel Parallelism in Computational Chemistry: Our Approach • Proposed solution to improve scalability • Increase granularity of computation => improve the overall scalability. • Exploitation of multiple levels of parallelism (MLP) • Instead of execution entire application on the full set of processors, assign parts of application to appropriately-sized subsets of processors • Many apps qualify • Challenge: Difficult to implement • Use advanced tools to address programming complexity • Common Component Architecture (CCA) • Global Arrays (GA) shared-memory programming model • Objective:To demonstrate how CCA and GA can be used together to address requirements of real scientific applications

4. CCA Driver QM Gradient Gradient Gradient Gradient Energy Energy Energy Energy Energy Energy Energy Energy Energy Energy Energy Energy Technology • Technologies for exploiting multiple level parallelism • Global Arrays (GA) shared-memory programming model • High level parallel data management abstractions • Common Component Architecture (CCA) • Component technology for HPC applications • Hiding complexity • Enables composition of software modules written in different languages and programming styles

5. Multiple Component Multiple DataModel • Introducing Multiple Component Multiple Data • i.e. multiple program multiple data (MPMD) model in context of CCA • instantiating components on subgroups of processors • create a dynamic environment to partition computational resources and manage them to execute the overall application effectively • Facilitate dynamic behavior of the application itself for example • Resizing processor groups based on memory requirements or scaling characteristics • swapping components based on numerical or computational performance

6. Numerical Hessian Example Hessian • Numerical Hessian Algorithm • determination of energy second derivatives through numerical differentiation of gradients, which may in turn be obtained from numerical differentiation of energies • Multiple gradient calculations • Each gradient has multiple energy calculations • limited scalability • Not effectively utilizing variable degrees of parallelism Gradient Energy

7. QM Gradient Energy Energy Energy Numerical Hessian Scalability - I Single Energy Calculation • Single energy calculation does not scale beyond 4 processes* • Two-level Parallelism • Native parallel code – Energy level • group-based energy calculations at gradient level • using GA processor groups

8. CCA Driver Energy NWChem_QM_0 cProps Energy ModelFactory QM Gradient Gradient Gradient Gradient Param Port Energy Energy Energy Energy Energy NWChem_QM_1 Energy cProps Energy Energy Energy Energy ModelFactory Energy MCMD Hessian Driver ModelFactory Param Port cProps Energy Energy Energy Energy Go Energy NWChem_QM_2 cProps Energy ModelFactory Energy Param Port Energy Energy NWChem_QM_n cProps Energy ModelFactory Param Port Energy Multilevel Parallelism • Combining SPMD and MPMD paradigms • MCMD – Multi Component Multiple Data • MPMD + Component • The MCMD Driver launches multiple instances of NWChem QM components on subsets of processors (CCA) • Each NWChem QM (gradient) component does multiple energy computations on subgroups (GA) Gradient Energy Energy Energy

9. QM_0 QM_0 QM_0 QM_0 cProps cProps cProps cProps Energy ModelFactory ModelFactory ModelFactory ModelFactory Energy Parameter Parameter Parameter Parameter Energy Energy Energy Energy MCMD Driver cProps ModelFactory Go Builder Energy Energy Energy Energy cProps Builder Service Energy Builder Energy Multiple Component Multiple Data (CCA’s MCMD Model) MCMD Driver • Create new components • Create processor groups • Assign processor groups to components • Connect components • Collect results Collect Results

10. Numerical Hessian Scalability - II • Three-level Parallelism • Energy-Level • Native parallel code • Gradient-Level • group-based single energy calculations using GA groups • Hessian Level • Task-based gradient calculations using CCA Application efficiency improved 10x times on 256 CPUs

11. Potential Applications Relevant To This Approach • Molecular Dynamics • Monte Carlo • Growth nucleation • Numerical Hessians • Vibrational spectra • Optimization techniques • Simulated annealing with local optimization • Nudged Elastic Band methods • Determine reaction path for kinetic rates • Trajectory simulations

12. MCMD Programming • Multi-level parallelism • Nested parallel decomposition • Possibly multiple levels of parallelism • Multiple parallel simulations are run concurrently in a coupled fashion, exchanging data at boundaries or perhaps even within volumes.

13. MCMD Services • Develop MCMD services to support MLP • Creating and management of processor groups • CCA Represenation for Groups id, membership • Mapping of component to groups and their coordination • Coordination of concurrent and nested SCMD/MCMD tasks • Communication between groups • Dynamic reconfiguration • Handling termination of processor groups, components • MCMD as a service or a component ?

14. Activities • Year 1: • Develop a model to express Multi-level parallelism through processor groups • Requirements gathering and design of flexible dynamic multi-level parallelism model • Coordinate & interact with other initiatives (ongoing) • Year 2 • Define a CCA Standard way of specifying and translating processor group membership and mapping between components • Year 3, 4, 5. • …

15. Implications of MCMD for CCA model • Model for Applications with Multi-Level Parallelism – Important • Process group abstraction – compatible with MPI, PVM, GA, GAS languages, HPCS languages (?) • MPI as default ? Group translators • How to address threaded components? OpenMP? Pthreads? Processor group for a threaded component? • Group-awareness to CCA and a CCA way of naming groups • i.e. multi-level parallelism at the CCA level/BuilderService

16. Implications of MCMD for CCA Implementations • Processor group management • Run-time configuration • At run-time, user should be able to blow-up connections, create components and assign groups • Swapping components, .. • Mapping communicators • Overlapping/Disjoint processor groups

17. Summary - Found MCMD Effective • Implemented a flexible, multi-level software architecture for computational chemistry applications • Exploits variable levels of parallelism • A order of magnitude of performance improvement • Hiding complexity and enabling better s/w composition • MCMD model has potential for addressing scalability in future large scale systems • More work is needed in CCA infrastructure and s/w to take advantage for larger class of apps • Facilitate dynamic groups • Make MCMD easier to adopt for apps