CCA Components for Accelerator Physics ComPASS SAP Project and Phase II Doe SBIR

1. CCA Components for Accelerator Physics ComPASS SAP Project and Phase II Doe SBIR Stefan Muszala, Tech-X Corp, Boulder, CO In collaboration with Jim Amundson (FNAL), Lois Curfman McInnes and Boyana Norris (ANL) Christine Roark, Roopa Pundaleeka, Peter Stoltz (Tech-X)

2. High-performance accelerator software should allow complex applications while promoting good software engineering practices • Software reuse and common interfaces • Ability to compose simulations • Portability • Mixed language programming interoperability • Performance analysis of composed simulations

3. Phase 1 recap 1) Design interfaces and implement components for TxPhysics 2) Build Synergia2 components and run an Electron Cloud simulation 3) Begin performance measurement and modeling • 4) Additionally, modifications to Bocca • Reference: Two-tiered Component Design and Performance Analysis of Synergia2 Accelerator Simulations, S. Muszala, J. Amundson, L. C. McInnes, and B. Norris, accepted for publication in the Proceedings of the 2009 Workshop on Component-Based High Performance Computing, to be held Nov 15-16, 2009, Portland, Oregon.

4. Next major milestones that build toward ECE simulations with CQoS • CQoS (Computational Quality of Service) for accelerator simulations: How, during runtime, can we make make sound choices for reliability, accuracy, and performance, taking into account the problem instance and computational environment? • Composition: select initial component implementations and configuration parameters • Reconfiguration: change parameters • Substitution: change implementations • 1) Design and implement CCA Synergia2 components for use on leadership-class supercomputers • Software infrastructure (getting this in place first) • Interfaces and components • 2) Explore performance models for parallel architectures networks and algorithms • Performance analysis and modeling • Build CQoS infrastructure • 3) Demonstrate and compose various Synergia2 CCA electron cloud component use cases • Compose ECE simulations • Apply CQoS

5. Building and porting software on Surveyor • Babel • CCA Tools • Synergia • VORPAL

6. Building babel on surveyorhttps://trac.mcs.anl.gov/projects/cca/wiki/babelbgp • Can configure with everything except java support, most (but not all) features are set by runtime/m4/llnl_cross_compiling.m4 • Relies on fixing libtool convenience scripts for builds (contrib/libtool_fix.sh) but in particular, for any shared library... find . -name libtool -exec sed -il -e 's|^archive_cmds="\(....\).*|archive_cmds="\1 \\$libobjs \\$deplibs \\$compiler_flags -qmkshrobj -G -e \\$soname -o \\$lib"|g' You need this for many non-babel libraries too! • Don't put -G in LDFLAGS – this causes binaries to break

7. Libtool script fixes to run regressiontests on surveyorhttps://trac.mcs.anl.gov/projects/cca/wiki/babelbgp • DON'T build or run from /scratch! • Relies on script fixing (both libtool and python runs): contrib/regression_fix.sh • Replaces 'exec' with 'cobalt-mpirun' & env. Variables • Replaces '$PYTHON' with 'cobalt-mpirun' & env. Variables

8. Running regression tests • Make sure PYTHONPATH points to numpy-1.3.0 for BG/P in config line • Submit tests to the queue: runC, runCxx, runF77, runF90: qsub -A cca-tools -t 15 -n 1 –mode script 'runAll.sh' RunPy2*: qsub -A cca-tools -t 15 -n 1 –mode script 'runPy2*'

9. Building ccaffeine on surveyor https://trac.mcs.anl.gov/projects/cca/wiki/ccafebgp • -rpath is not compatible with IBM compilers; always use -Wl,-rpath • IBM compilers also do not appreciate –Wall • setup.py needs to be handled differently: - Boyana has built a patched distutils • Build needs to be staged: Complete build and then rebuild drivers, to link them to the right place • Don't use babel-libtool! It causes improper linking

10. Final Notes, BG/P and CCA tools • Just because it builds, that doesn't mean it runs! surveyor:/home/projects/ccatools/cca/ccafe/bin> ./ccafe-single ./ccafe-single: error while loading shared libraries: ./.libs/libccafePreload_0_8_8.so: cannot open shared object file: No such file or directory • Libtool and IBM compilers don't always play nice together when linking against uninstalled libraries • Be wary of gcc compiler directives beginning with '-r' • Know that static libraries installed for BG/P are not position-independent • Never use babel-libtool unless it’s for Babel

11. Synergia Build f • --Testing Build. Combining Stephen and Jim A. instructions for a • “cut and paste” build • https://ice.txcorp.com/trac/ccaEcloud/wiki • -- The usual suspects: Boost and Chef • -- Prototype implementation for adding CTEST and CDASH to Chef to • implement full testing system

12. VORPAL Build • VORPAL Builds and mostly runs • Problem with solve that uses Trilinos • Stencil solve slower than original solve…hmmm • In the process of making equivalent Synergia2 and • VORPAL simulations

13. VORPAL BG/P and AMD solve performance Time (Sec) Number Processors

14. Babel and Fortran Struct work continues (My focus for the month of February) -- F90 redo: - F90 Server introduces a Fortran “fSkel” file. - Structs are unrolled in C, passed as individual arguments then repacked in fSkel. F90 Impl stays the same. - If you want zero-copy structs use F2003 binding. -- F77 – Needs clients, servers done -- Combine struct BindC and struct regression tests -- Added nightly runs to 32 and 64 bit AMD and 64 bit Intel with GCC and PGI

16. BlueGene/P system architecture • surveyor is a 1-rack (1024 CPU/4096 core) BG/P • PPC 970MP frontend login nodes