A New Paradigm for Large-scale Science: Computational End Stations

1. A New Paradigm for Large-scale Science:Computational End Stations Al Geist Computer Science and Mathematics Division Oak Ridge National Laboratory CCSG Conference Lyon France September 27, 2004

2. Neutron Reflectometer NERSC 3,328-processor 5 teraflop/s Ultra-high vacuum station Sample ORNL-CCS 1.08 Teraflop/s5 Teraflop/s-FY2002 Materials Science Virtual User Center Materials : Math : Computer Scientists • Open Source Repository • Object Oriented Tool Kit • Workshops • Education How it all got started: Science facility analogy User Community Facility End Station • NERSC • 3,328-processor • 5 teraflop/s Fe User Community From a presentation by Stocks and Harmon in December 2002 to the DoE Council on Materials Science and Engineering, led by Peter Flynn and David Mermin

3. Pioneer the concept ofComputational End Stations for user access High-end Computing Revitalization Task Force 2004 • Must focus on simulations not possible to do otherwise • Let Grids and capacity computers handle capacity problems Deploy a fundamentally new approach for long-term engagement of research communities modeled on the “end station” concept through which major experimental facilities provide specialized instruments to specific user groups HECRTFWorkshop Report “Establishing computational end stations is one integrated solutionto the needs of developers and users. … Application codes and their associated analysis tools are the instruments of computational science.” Treat capability computing as valuable, scarce resource

4. How End Stations work at Large Experimental Facilities (or Why would groups do this?) End Station

5. National Leadership Computing Facility DOE Proposal awarded July 2004 Leadership class computing is critical for scientific and technological innovation • Not a simple notion of being number #1 on Top500 list • Requires sustained commitment to support the computational needs of science • Infrastructure to generate breakthrough science and continuous technological innovation Unique features of this Facility • Operate as a capability computing center (big users only) • Focus on computationally intensive projects of large scale and high scientific impact through competitive peer review process • Computational end stations

6. New ORNL facility capable of housing petascale class computers ORNL’s Computational Science Complex • Space and power: • 40,000 ft2 computer room with 36-in. raised floor, 18 ft deck to deck • 8 MW of power (expandable) @ 5c/kWhr • 450 offices, 14 conference rooms • Large visualization center w/ 30’ Powerwall • Separate lab areas for quantum computing, cluster, and network research

7. NLCF Hardware Roadmap No one architecture solution is right for all application needs Cray X1/X2 1000 TF Vector Arch Global mem. Powerful CPU Cray X1e Cray RS IBM BG 250 TF Cray X2 100 TF Cluster Arch Low latency High bandwidth 50 TF 40 TF 5 TF IBM Blue Gene Scalability 100K CPU MB/CPU 2009 2004 2005 2006 2007 2008

8. Science teams enabled through “End Stations” Open End Station CapabilityPlatform Development End Station Ultrascale Hardware HW teams End station 2 Research team End station 1 High-End science problem Tuned code Computational End Stations Software & Libs SW teams BreakthroughScience

9. Two special end stations Open End Station: For use by individuals or groups who have important scientific problem that requires NLCF resources but doesn’t fit existing end stations. Proposals are peer-reviewed by “open end station” committee Development End Station: For use by groups developing software for NLCF resources with goal to submit a proposal for future NLCF end station. Also used by CS teams developing tuned libraries and runtime software for use across all end stations.

10. Six to Ten Computational End Stations • End Station defined by three characteristics: • Addresses problem area of national importance (eg. nanotech) and requires the unique resources of facility • Scientific team willing to create and maintain end station • Suite of scientific codes in area tuned to the NLCF resources National Problem Scientific team application suite

11. Selection ofNLCF Computational End Stations Draft Selection process open to all scientists and engineers nationwide : • Call for end station proposals (nominally for 5 yr lifetime but renewable) • NLCF end station proposals are reviewed by panel of nationally recognized scientists • Selected teams allocated portion of NLCF resources, but must raise funds to build, maintain, and evolve end station • Once the end station is created: • NLCF director appoints peer review panel to review proposals for external use of end station • NLCF issues call for proposals to use end station • NLCF annually reviews end stations WRT science breakthroughs and community use CES proposals CES teams Peer review Select users Science results

12. Proposed End Stations International teams are being formed to create NLCF “end stations” around the following areas. Design of innovativenano-materials with desired properties Nano Bio Climate Fusion Predictive understanding of microbial molecular and cellular systems 100 yr Global climate to support policy decisions Predictive simulations of full scale devices ITER Time is right to get involved on end station teams There are also end station users (scientific proposals)

13. Computational Climate End StationChief Scientist: Warren Washington (National Center Atmospheric Research)Instrument Scientist: John Drake Building the Instrument: Experiments: • For simulations that improve scientific basis of models • For climate change simulations that address national concerns and contribute to DOE Science mission (e.g. regional climate change, carbon and biogeochemical cycles) • Experiment proposals peer reviewed by partners (PRP) • http://nlcf.ornl.gov/climateces • Based on CCSM and supporting frameworks • Instrument Team overlaps SciDAC CCSM Consortium and NCAR SE Group • Remote access/analysis through Earth System Grid (ESG) • Partners are NCAR, ORNL, PCMDI, COSIM, ANL, PNNL, NASA-Goddard, GaTech, Duke

14. (b) Micro - (c) Extended - (d) Transport Codes turbulence codes (a) RF codes MHD codes Fusion Computational End station o Multiscale software suite

15. Molecular Dynamics Electronic Structure (DFT) Classical Monte Carlo Finite Element Quantum Monte Carlo XML based I/O system Lattice Boltzmann Various Quantum Many- Body (…) Nanoscience Computational End Station Quantum models for strongly correlated systems High temperature superconductivity Colossal magnetoresistance (CMR) Bose-Einstein vs. BCS condensates Electronic structure methods for nanostructures Free energy surfaces of magnetic nano-particles Quantum transport, spin dependent tunneling Molecular dynamics simulations Nano-tribology Protein structure and transcription factor binding sites Software Tools Suite

16. Software and software engineering issues for End Stations • Software library needs are similar to those of the Climate community • Good F95, C, C++ compilers/debuggers/optimizers, BLAS, LAPACK, ScaLAPACK, NAG, IMSL, etc. • Good sparse matrix solvers - need efficient ports of libraries such as PETSc, MUMPS • Matrices are often ill-conditioned due to large scale separations, anisotropy • End station should have access to several individuals well-versed in the Cray architecture • Need to be involved early on in code structuring projects • Tools for variable blocking length for portability between vector and superscalar systems • Checkpoint/restart capability

17. Final Thoughts • Computer lifetime is 3-5 Years but it takes decades to develop methods and software • NLCF hardware is a moving target! • Nanoscience continues to be an emerging field covering many domains in materials, condensed matter, and biological sciences • Computational challenges are a moving target! Ergo, a robust software architecture optimized for rapid evolution, as well as performance, is essential to achieve effective Computational End Stations.