Rusty Lusk Mathematics and Computer Science Division Argonne National Laboratory

1. One Language to Rule Them All or ADA Strikes Back?-- An Update on the DARPA HPCS Language Project Rusty Lusk Mathematics and Computer Science Division Argonne National Laboratory

2. Outline • The DARPA HPCS language project • Why it might not work • This has been tried before… • The “local maximum” problem • Why it might work • The HPCS languages as a group • The Plan

3. The Lay of the Land in Scalable Parallel Programming • The current standard: Fortran-77, Fortran-90 (2003), C, C++ programs with calls to MPI. Use of MPI-2 increasing, especially for I/O. Message-passing model. • The PGAS (Partitioned Global Address Space) languages: UPC, Co-Array Fortran, and Titanium. Each implements a similar programming model inside a different base language (C, Fortran-90, and Java, respectively). Static number of processes. Global view of data, but explicit local/remote distinction for data for performance. (2nd Annual PGAS Conference, Oct 3-4, Washington) • The HPCS languages: No fixed number of processes. Many advanced features. Global view of data, with locality hints. • http://crd.lbl.gov/~parry/hpcs_resources.html

4. The DARPA HPCS Language Project (in a nutshell) • The DARPA High Productivity Computer Systems (HPCS) Project is a 10-year, three-phase, hardware/software effort to transform the productivity aspect of the HPC enterprise. • In Phase II, just ending, three vendors were funded to develop high productivity language systems, and they are working on it. • IBM: X10 • Cray: Chapel • Sun: Fortress

5. DARPA HPCS Language Project (continued) • These languages are expected to run well on and exploit the HPCS hardware platforms being developed by all three vendors (soon to be less than three). • But it is recognized that, in order to be adopted, any HPCS language will also have to be effective on other parallel architectures. • (And, ironically, the HPCS hardware will have to run Fortran + MPI programs well.) • DARPA has also recently been funding a small academic effort (at Argonne (Gropp & Lusk), Berkeley (Yelick), Rice (Mellor-Crummey), and Oak Ridge (Harrison) • Connect the HPCS languages to research in compilers and runtime systems currently being done for the PGAS languages and MPI • Promote the eventual convergence of the language efforts into a single language.

6. “HPCS” languages have been tried before… • The Japanese 5th generation project had many similarities and near similarities to the DARPA program • 10 year program • A new parallel language (Concurrent Logic Programming) expressing a new programming model • New language presented as more productive • Multiple vendors, each implementing hardware support for the language, to address performance problems. • ADA was intended to replace both Fortran and Cobol. It included built-in parallelism. It was successful in its way: many (defense) applications were implemented in ADA. A large community of ADA programmers was created. • Neither language is with us today. It is not clear whether either even had any influence on our current parallel programming landscape.

7. Why It Might Not Work – Part 1: The Problem is Hard • Programmers do value productivity, but reserve the right to define it. • Past experience says that ambitious new languages are a risky business • The HPCS languages have some research issues in their paths • These languages will need not only heroic compilers but heroic runtimes • Performance not seriously addressed yet • Applications not inherent part of language design teams • More later on how these problems are being addressed

8. Why It Might Not Work – Part 2:the Competition Is Tough • MPI represents a very complete definition of a well-defined programming model • There are many implementations • Vendors • Open source • Enables high performance for wide class of architectures • Small subset easy to learn and use • Expert MPI programmers needed most for libraries, which are encouraged by the MPI design.

9. Why Is It So Hard to Get Past MPI?(Important to understand when contemplating alternatives) • Open process of definition • All invited, but hard work required • All drafts and deliberations open at all times • First decision was not to standardize on any existing system • Portability • Need not lead to lowest common denominator approach • MPI semantics allow aggressive implementations • Ubiquitous; applications can be developed anywhere • Performance • MPI can help manage the memory hierarchy • Collective operations can provide scalability • Cooperates with optimizing compilers • Simplicity • MPI (i. e., MPI-2) has 275 functions; is that a lot? (MFC) • Can write serious MPI applications with 6 functions • Economy of concepts, for example communicators, datatypes

10. Why Is It So Hard to Get Past MPI?(continued) • Modularity • MPI supports component-based software with communicators • Support for libraries means some applications may contain no MPI calls • Composability • MPI works with other tools (compilers, debuggers, profilers) • Provides precise interactions with multithreaded programs • Completeness • Any parallel algorithm can be expressed • Easy things are not always easy with MPI, but • Hard things are possible • Transparency • What you code is what you get • Implementation research continues in both vendor and open-source communities.

11. MPICH2 – 2nd-generation MPI-2 Implementation • Both research and software • Focus on performance, completeness, scalability • Algorithms for collective operations • New algorithms for handling datatypes • Collaborations with HPC vendors • Recent MPICH2 work has focused on improving low-level runtime interface and its implementation • This impacts all the portability variations of MPICH2

12. MPI Latency for Shared Memory

13. Shared Memory Bandwidth

14. Why It Might Work (Why HPCS Languages Might Have an Impact) • There is widespread interest in the parallel programming issue, and MPI can be painful. • There is a transition path through the PGAS languages, where significant compiler and runtime research has been done. • There is a plan. • The vendors have all done very interesting work.

15. Looking at the HPCS Languages as a Group • Base Language • Creation of parallelism • Communication and/or data sharing • Synchronization among threads/processes • Locality (associating computation with data)

16. Base Language • The languages use different sequential bases • X10 uses an existing OO language, Java • Inherits good and bad features • Adds support for arrays, value types, parallelism • Gains tool support from Java environment • Chapel and Fortress use own OO language • Can tailor it to science (Fortress explores new character sets) • Large intellectual effort in getting base right • No specific analysis of base language • Question: How large a step can a new language be? • Successful new languages have been small steps • C, C++, Java

17. Creation of Parallelism • All three HPCS languages have parallel semantics • Not just data parallel with serial semantics • No reliance on automatic parallelism • All have: • Dynamic parallelism for loops as well as tasks • Mixed task and data parallelism • Encouragement for programmer to specify as much parallelism as possible, with the idea that the compiler/runtime will control how much is actually executed in parallel

18. Sharing and Communication • Chapel and Fortress are Global Address Space Languages • Can directly read/write remote variables • Similar to PGAS languages • X10 is a “parallel OO” language • Remote execution of methods • All are implicitly asynchronous • Sequential consistency within a “place”. • No collective communication. • Distributed arrays • In Chapel distribution is separate annotation • In X10 no direct remote access for distributed array data • In Fortress user-defined distributed data structures without explicit layout control

19. Synchronization • All three languages support atomic blocks • None of the languages have locks • (Semantically, locks are more powerful, but harder to manage than atomic sections) • Other mechanisms • X10 has “clocks” (barriers with dynamically attached tasks), conditional atomic sections, synchronization variables • Chapel has “single” (single writer) and “sync” (multiple readers and writers) variables) • Fortress has abortable atomic sections and a mechanism for waiting on individual spawned threads.

20. Locality • All three languages have a way to associate computation with data, for performance • It looks a little different in each language (“places” vs. “locales”) • Explicitly distributed data structures enable automation of this • Especially for arrays • Delegation of the problem to libraries, esp. in Fortress

21. Recent Activities on the HPCS Front • The HPCS/PGAS project has resulted in collaboration on several compiler/runtime issues with the vendors • Workshop at Oak Ridge in July, 2006 for assessment, comparison, and future planning • Tentative plan made for near- and medium-term developments • Incorporation into the larger HPCS productivity plan

22. Thoughts from the Recent Workshop • A main purpose of the workshop was to explicitly explore the topic of converging the HPCS languages to a single language. • Perhaps life would be tidier if there were only one HPCS language being discussed at this point. • But it would be less interesting. • And perhaps less effective in the long run. • Premature convergence might be a bad idea. • Some technical • Some philosophical • Diversity is good • One summary of the current situation is that there is lots of diversity • Challenge: how to make sure this remains good

23. Diversity in the Vendor Approaches • The HPCS vendors are not just designing three different versions of the same type of thing • Chapel: “To boldly go where no programming language has gone before” • X10: Extending existing language environment, fitting into larger structure • Fortress: Providing a framework for parallel language design; taking risks with visual aspects of programming text • It would be a shame to lose any of these points of view

24. But Some Uniformity in Things Missing So Far • Language support for high-performance I/O • Coalescing through collectivity • Tackling performance • Necessary in order to attract attention from applications

25. Diversity in Applications • One extreme: “Current practice is perceived as working well” • MPI being used as intended • The other: Cannot do critical applications at all without new way of programming. • More dynamism, expressivity needed • In between: No one is against productivity, but new approaches have to be strictly better than current ones to justify investment in change.

26. Diversity in Relevant CS Research • PGAS Languages – UPC, Co-Array Fortran, Titanium • Language design • Compilation techniques applicable to HPCS languages • Runtime system requirements • Deep collaboration with applications • Happening with PGAS, not yet with HPCS • Runtime system implementation • Could this level be standardized? • Semantics • Syntax • Would make it easier for computer scientists to design new languages, which is not necessarily a bad thing.

27. Some Paths Forward • Completion of current language design projects • Focus on performance credibility, at least for part of language • The UPC experience • Convergence • At all? • On What? • When? • Some lessons from a prior convergence effort • Could we carry out an MPI forum – like process?

28. The MPI Forum • A convergence process to produce portability among message-passing libraries • Regarded as successful in meeting its original goals • But not certain to succeed as it went along • Standardized some new ideas along the way • Message-passing libraries already existed • From vendors: mpl, eui, cmmd, nx, ncube, meiko, others • Open source: pvm, p4, others • They had similar semantics • There was application experience with all of these. • Vendors were pre-committed to adopting the MPI Forum result in their mainstream parallel computer products. • Decided up front not to pick an existing system • An implementation tracked the standard development.

29. Could Such a Process be Used In the HPCS Language Project? • It depends on what and when. • Certainly now is too early to try to produce a single language. • Perhaps it could work with a common runtime system, if everyone participated

30. A Proposed Schedule • Next 18 months: Vendors try to achieve: • “Frozen” version of syntax • Some workshops on common technical implementation issues • Preliminary work on common runtime system by CS researchers and vendors • Performance demo of some subset on some parallel machine • Next three years • Vendors improve performance • Applications get experience • Some evolution inevitable as implementers get experience • New common HPCS language design effort begins in 2010-11 • The shared experiences of the preceding three years make this go smoothly, and vendors track design with implementations. • 2013 • All applications become happily productive and remain so forever.

31. Some Possible Workshop Topics • Distributions • Memory models • Arrays and regions • Locales, places, etc. • Task parallelism • Common runtime • Synchronization • Atomic transactions • Parallel I/O embedded in the language • Types • Interactions with other languages, libraries • Applications • Tools for HPCS languages explcitly, especially debuggers • Just starting to plan some of these

32. DARPA Productivity Team Phase III

33. Summary • The DARPA HPCS language project is active and healthy • Excellent work is being done by the vendors • It is time to get the larger community involved • Getting a new language, much less a new paradigm is an admitted challenge • The current situation is not all that bad, which doesn’t help. • There is a reasonable plan for moving forward at a measured pace. • There are a lot of languages…