UPC at NERSC/LBNL

1. UPC at NERSC/LBNL Kathy Yelick, Christian Bell, Dan Bonachea, Yannick Cote, Jason Duell, Paul Hargrove, Parry Husbands, Costin Iancu, Mike Welcome NERSC, U.C. Berkeley, and Concordia U.

2. Overview of NERSC Effort Three components: • Compilers • IBM SP platform and PC clusters are main targets • Portable compiler infrasturucture (UPC->C) • Optimization of communication and global pointers • Runtime systems for multiple compilers • Allow use by other languages (Titanium and CAF) • And in other UPC compilers • Performance evaluation • Applications and benchmarks • Currently looking at NAS PB • Evaluating language and compilers • Plan to do a larger application next year

3. NERSC UPC Compiler • Compiler being developed by Costin Iancu • Based on Open64 compiler for C • Originally developed at SGI • Has IA64 backend with some ongoing development • Software available on SourceForge • Can use as C to C translator • Can either generate before most optimizations • Or after, but this is known to be buggy right now • Status • Parses and type-checks UPC • Finishing code generation for UPC->C translator • Code generation for SMPs underway

4. Compiler Optimizations • Based on lessons learned from • Titanium: UPC in Java • Split-C: one of the UPC predecessors • Optimizations • Pointer optimizations: • Optimization of phase-less pointers • Turn global pointers into local ones • Overlap • Split-phase • Merge “synchs” at barrier • Aggregation Split-C data on CM-5

5. Portable Runtime Support • Developing a runtime layer that can be easily ported and tuned to multiple architectures. Direct implementations of parts of full GASNet UPCNet: Global pointers (opaque type with rich set of pointer operations), memory management, job startup, etc. Generic support for UPC, CAF, Titanium GASNet Extended API: Supports put, get, locks, barrier, bulk, scatter/gather GASNet Core API: Small interface based on “Active Messages” Core sufficient for functional implementation

6. Portable Runtime Support • Full runtime designed to be used by multiple compilers • NERSC compiler based on Open64 • Intrepid compiler based on gcc • Communication layer designed to run on multiple machines • Hardware shared memory (direct load/store) • IBM SP (LAPI) • Myrinet 2K (GM) • Quadrics (Elan3) • Dolphin • VIA and Infiniband in anticipation of future networks • MPI for portability • Use communication micro-benchmarks to choose optimizations

7. Possible Optimizations • Use of lightweight communication • Converting reads to writes (or reverse) • Overlapping communication with communication • Overlapping communication with computation • Aggregating small messages into larger ones

8. MPI vs. LAPI on the IBM SP • LAPI generally faster than MPI • Non-Blocking (relaxed) faster than blocking

9. Overlapping Computation: IBM SP • Nearly all software overhead – no computation overlap • Recall: 36 usec blocking, 12 usec nonblocking

10. Conclusions for IBM SP • LAPI is better the MPI • Reads/Writes roughly the same cost • Overlapping communication with communication (pipelining) is important • Overlapping communication with computation • Important if no communication overlap • Minimal value if >= 2 messages overlapped • Large messages are still much more efficient • Generally noisy data: hard to control

11. Other Machines • Observations: • Low latency reveals programming advantage • T3E is still much better than the other networks usec

12. Applications Status • Short term goal: • Evaluate language and compilers using small applications • Longer term, identify large application • Conjugate Gradient • Show advantage of t3e network model and UPC • Performance on Compaq machine worse: • Serial code • Communication performance • Simple n2 particle simulation • Currently working on NAS MG • Need for shared array arithmetic optimizations

13. Future Plans • This month • Draft of runtime spec • Draft of GASNet spec • This year • Initial runtime implementation on shared memory • Runtime implementation on distributed memory (M2K, SP) • NERSC compiler release 1.0b for IBM SP • Next year • Compiler release for PC cluster • Development of CLUMP compiler • Begin large application effort • More GASNet implementations • Advanced analysis and optimizations

14. Runtime Breakout • How many runtime systems? • Compaq MTU • LBNL/Intrepid • Language issues • Locks • Richard Stallman’s ? • upc_phaseof for pointers with indef. block size • Misc • Runtime extensions • Strided and scatter/gather memcopy

15. Read/Write Behavior • Negligible difference between blocking read and write performance

16. Overlapping Communication • Effects of pipelined communication are significant • 8 overlapped messages are sufficient to saturate NI Queue depth

17. Overlapping Computation • Same experiment, but fix total amount of computation

18. SPMV on Compaq/Quadrics • Seeing 15 usec latency for small msgs • Data for 1 thread per node

19. Optimization Strategy • Optimizations of communication is key to making UPC more usable • Two problems: • Analysis of code to determine which optimizations are legal • Use of performance models to select transformations to improve performance • Focus on the second problem here

20. Runtime Status • Characterizing network performance • Low latency (low overhead) -> programmability • Specification of portable runtime • Communication layer (UPC, Titanium, Co-Array Fortran) • Built on small “core” layer; interoperability a major concern • Full runtime has memory management, job startup, etc. usec

21. What is UPC? • UPC is an explicitly parallel language • Global address space; can read/write remote memory • Programmer control over layout and scheduling • From Split-C, AC, PCP • Why a new language? • Easier to use than MPI, especially for program with complicated data structures • Possibly faster on some machines, but current goal is comparable performance p0 p1 p2

22. Background • UPC efforts elsewhere • IDA: t3e implementation based on old gcc • GMU (documentation) and UMC (benchmarking) • Compaq (Alpha cluster and C+MPI compiler (with MTU)) • Cray, Sun, and HP (implementations) • Intrepid (SGI compiler and t3e compiler) • UPC Book: • T. El-Ghazawi, B. Carlson, T. Sterling, K. Yelick • Three components of NERSC effort • Compilers (SP and PC clusters) + optimization (DOE/UPC) • Runtime systems for multiple compilers (DOE/Pmodels + NSA) • Applications and benchmarks (DOE/UPC)

23. Overlapping Computation on Quadrics 8-Byte non-blocking put on Compaq/Quadrics