1 / 32

Portable MPI and Related Parallel Development Tools

Explore the world of MPI, a message-passing library specification, along with its implementation, portability, and performance. Learn about tools and libraries for parallel programming and future developments in this field.

heinea
Download Presentation

Portable MPI and Related Parallel Development Tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Portable MPI and Related Parallel Development Tools Rusty Lusk Mathematics and Computer Science Division Argonne National Laboratory (The rest of our group: Bill Gropp, Rob Ross, David Ashton, Brian Toonen, Anthony Chan)

  2. Outline • MPI • What is it? • Where did it come from? • One implementation • Why has it “succeeded”? • Case study: an MPI application • Portability • Libraries • Tools • Future developments in parallel programming • MPI development • Languages • Speculative approaches

  3. What is MPI? • A message-passing library specification • extended message-passing model • not a language or compiler specification • not a specific implementation or product • For parallel computers, clusters, and heterogeneous networks • Full-featured • Designed to provide access to advanced parallel hardware for • end users • library writers • tool developers

  4. Where Did MPI Come From? • Early vendor systems (NX, EUI, CMMD) were not portable. • Early portable systems (PVM, p4, TCGMSG, Chameleon) were mainly research efforts. • Did not address the full spectrum of message-passing issues • Lacked vendor support • Were not implemented at the most efficient level • The MPI Forum organized in 1992 with broad participation by vendors, library writers, and end users. • MPI Standard (1.0) released June, 1994; many implementation efforts. • MPI-2 Standard (1.2 and 2.0) released July, 1997.

  5. Informal Status Assessment • All MPP vendors now have MPI-1. (1.0, 1.1, or 1.2) • Public implementations (MPICH, LAM, CHIMP) support heterogeneous workstation networks. • MPI-2 implementations are being undertaken now by all vendors. • MPI-2 is harder to implement than MPI-1 was. • MPI-2 implementations will appear piecemeal, with I/O first.

  6. MPI Sources • The Standard itself: • at http://www.mpi-forum.org • All MPI official releases, in both postscript and HTML • Books on MPI and MPI-2: • Using MPI: Portable Parallel Programming with the Message-Passing Interface (2nd edition), by Gropp, Lusk, and Skjellum, MIT Press, 1999. • Using MPI-2: Extending the Message-Passing Interface, by Gropp, Lusk, and Thakur, MIT Press, 1999 • MPI: The Complete Reference, volumes 1 and 2, MIT Press, 1999. • Other information on Web: • at http://www.mcs.anl.gov/mpi • pointers to lots of stuff, including other talks and tutorials, a FAQ, other MPI pages

  7. The MPI Standard Documentation

  8. Tutorial Material on MPI, MPI-2

  9. The MPICH Implementation of MPI • As a research project: exploring tradeoffs between performance and portability; conducting research in implementation issues. • As a software project: providing a free MPI implementation on most machines; enabling vendors and others to build complete MPI implementation on their own communication services. • MPICH 1.2.2 just released, with complete MPI-1, parts of MPI-2 (I/O and C++), port to Windows2000. • Available at http://www.mcs.anl.gov/mpi/mpich

  10. Lessons From MPIWhy Has It Succeeded? • The MPI Process • Portability • Performance • Simplicity • Modularity • Composability • Completeness

  11. The MPI Process • Started with open invitation to all those interested in standardizing message-passing model • Participation from • Parallel computing vendors • Computer Scientists • Application scientists • Open process • All invited, but hard work required • All deliberations available at all times • Reference implementation developed during design process • Helped debug design • Immediately available when design completed

  12. Portability • Most important property of a programming model for high-performance computing • Application lifetimes 5 to 20 years • Hardware lifetimes much shorter • (not to mention corporate lifetimes!) • Need not lead to lowest common denominator approach • Example: MPI semantics allow direct copy of data from user space send buffer to user space receive buffer • Might be implemented by hardware data mover • Might be implemented by netwrk hardware • Might be implemented by socket • The hard part: portability with performance

  13. Performance • MPI can help manage the crucial memory hierarchy • Local vs. remote memory is explicit • A received message is likely to be in cache • MPI provides collective operations for both communication and computation that hide complexity or non-portability of scalable algorithms from the programmer. • Can interoperate with optimising compilers • Promotes use of high-performance libraries • Doesn’t provide performance portability • This problem is still too hard, even for the best compilers • E.g. BLAS

  14. Simplicity • Simplicity is in the eye of the beholder • MPI-1 has about 125 functions • Too big! • Too small! • MPI-2 has about 150 more • Even this is not very many by comparison • Few applications use all of MPI • But few MPI functions go unused • One can write serious MPI programs with as few as six functions • Other programs with a different six… • Economy of concepts • Communicators encapsulate both process groups and contexts • Datatypes both enable heterogeneous communication and allow non-contiguous messages buffers • Symmetry helps make MPI easy to understand.

  15. Modularity • Modern applications often combine multiple parallel components. • MPI supports component-oriented software through its use of communicators • Support of libraries means applications may contain no MPI calls at all.

  16. Composability • MPI works with other tools • Compilers • Since it is a library • Debuggers • Debugging interface used by MPICH, TotalView, others • Profiling tools • The MPI “profiling interface” is part of standard • MPI-2 provides precise interaction with multi-threaded programs • MPI_THREAD_SINGLE • MPI_THREAD_FUNNELLED (OpenMP loops) • MPI_THREAD_SERIAL (Open MP single) • MPI_THREAD_MULTIPLE • The interface provides for both portability and performance

  17. Completeness • MPI provides a complete programming model. • Any parallel algorithm can be expressed. • Collective operations operate on subsets of processes. • Easy things are not always easy, but • Hard things are possible.

  18. The Center for the Study of Astrophysical Thermonuclear Flashes • To simulate matter accumulation on the surface of compact stars, nuclear ignition of the accreted (and possibly underlying stellar) material, and the subsequent evolution of the star’s interior, surface, and exterior • X-ray bursts (on neutron star surfaces) • Novae (on white dwarf surfaces) • Type Ia supernovae (in white dwarf interiors)

  19. FLASH Scientific Results • Wide range of compressibility • Wide range of length and time scales • Many interacting physical processes • Only indirect validation possible • Rapidly evolving computing environment • Many people in collaboration Flame-vortex interactions Compressible turbulence Laser-driven shock instabilities Nova outbursts on white dwarfs Richtmyer-Meshkov instability Cellular detonations Helium burning on neutron stars Rayleigh-Taylor instability Gordon Bell prize at SC2000

  20. The FLASH Code: MPI in Action • Solves complex systems of equations for hydrodynamics and nuclear burning • Written primarily in Fortran-90 • Uses Paramesh library for adaptive mesh refinement; Paramesh is implemented with MPI • I/O (for checkpointing, visualization, other purposes) done with HDF-5 library, which is implemented with MPI-IO • Debugged with TotalView, using standard debugger interface • Tuned with Jumpshot and Vampir, using MPI profiling interface • Gordon Bell prize winner in 2000 • Portable to all parallel computing environments (since MPI)

  21. FLASH Scaling Runs

  22. X-Ray Burst on the Surface of a Neutron Star

  23. Showing the AMR Grid

  24. Processes Logfile Jumpshot Display MPI Performance Visualization with Jumpshot • For detailed analysis of parallel program behavior, timestamped events are collected into a log file during the run. • A separate display program (Jumpshot) aids the user in conducting a post mortem analysis of program behavior. • Log files can become large, making it impossible to inspect the entire program at once. • The FLASH Project motivated an indexed file format (SLOG) that uses a preview to select a time of interest and quickly display an interval.

  25. Removing Barriers From Paramesh

  26. Using Jumpshot • MPI functions and messages automatically logged • User-defined states • Nested states • Zooming and scrolling • Spotting opportunities for optimization

  27. Future Developments in Parallel Programming: MPI and Beyond • MPI not perfect • Any widely-used replacement will have to share the properties that made MPI a success. • Some directions (in decreasing order of speculativeness) • Improvements to MPI implementations • Improvements to the MPI definition • Continued evolution of libraries • Research and development for parallel languages • Further out: radically different programming models for radically different architectures.

  28. MPI Implementations • Implementations beget implementation research • Datatypes, I/O, memory motion elimination • On most platforms, better collective • Most MPI implementations build collective on point-to-point, too high-level • Need stream-oriented methods that understand MPI datatypes • Optimize for new hardware • In progress for VIA, Infiniband • Need more emphasis on collective operations • Off-loading message processing onto NIC • Scaling beyond 10,000 processes • Parallel I/O • Clusters • Remote • Fault-tolerance • Intercommunicators provide an approach • Working with multithreading approaches

  29. Improvements to MPI Itself • Better Remote-memory-access interface • Simpler for some simple operations • Atomic fetch-and-increment • Some minor fixup already in progress • MPI 2.1 • Building on experience with MPI-2 • Interactions with compilers

  30. Libraries and Languages • General Libraries • Global Arrays • PETSc • ScaLAPACK • Application-specific libraries • Most built on MPI, at least for portable version.

  31. More Speculative Approaches • HTMT for Petaflops • Blue Gene • PIMS • MTA • All will need a programming model that explicitly manages a deep memory hierarchy. • Exotic + small benefit = dead

  32. Summary • MPI is a successful example of a community defining, implementing, and adopting a standard programming methodology. • It happened because of the open MPI process, the MPI design itself, and early implementation. • MPI research continues to refine implementations on modern platforms, and this is the “main road” ahead. • Tools that work with MPI programs are thus a good investment. • MPI provides portability and performance for complex applications on a variety of architectures.

More Related