Results from the fusion working group of the Exascale Workshop

1. Results from the fusion working group of the Exascale Workshop Fusion Working Group Yasuhiro Idomura & Phil Ferguson, co-chairs David Bernholdt Sophie Blondel John Canik C S Chang David Green Shinay Maeyama John Mandrekas Masanori Nunami Andreas Wingen September 6, 2014

2. How do we enable fusion energy science to develop applications scalable to the exascale using co-design techniques? • The scalability of fusion energy science applications can be sensitive to the network topology • Japanese experience on the K computer indicates that a periodic torus network resulted in better scalability for gyrokinetics codes • A more unified memory structure would be easier for developers • Develop standard interfaces for exchange of information to allow interoperability of different exascale applications • Scalable libraries • Development of resilient software and hardware • Tools that enable software development to be portable • abstract machine model, programming models, code generation • Mini-apps sound reasonable. But there is concern about the sustainability and capturing the true physics of the problems.

3. How do we enable fusion energy science to develop applications scalable to the exascale using co-design techniques? • Assistance with algorithms, especially time parallelization • “From scratch” reformulations may be required • Coupling methodologies, including discrete ↔ continuum • Accept that every code will not be exascale

4. Given scalable exascale applications, what scientific outcomes would be expected in 2020+ timeframe? • ITER operations supported by first principles based, multiphysics burning plasma simulations • including core and edge plasmas, including L-mode to H-mode transition • Routine application of 5D gyrokinetic code(s) to the whole volume ITER plasma, including some 6D domains • Whole device modeling, including ITER relevant physics such as energetic particles • Some physics modules may still need to be provided externally: e.g., RF wave modules • RF modeling for full 3D ITER device scale, full-f PIC for RF • Predictive simulation of disruption avoidance and mitigation and Edge Localized Modes (ELMs) • Good qualitative, and semi-quantitative understanding of plasma-material interactions

5. What new emerging fields do you see in the exascale era? • First principles based plasma simulations routinely used for engineering design of actuators and operation scenarios • Start of 6D Vlasov Fokker-Planck simulation of tokamak plasmas in a limited scope • “Real-time” experiment interpretation and analysis • Automatic generation of error bounds and numerical uncertainties

6. Outline a plan for sustaining these teams and collaborations through 2020+ • Exchange information on latest Peta-scale algorithms, libraries, etc. • Explore new algorithms and approaches for exascale architectures • Build multidisciplinary teams modeled upon the SciDACframework as much as possible • Continue and build upon the existing collaborations and exchanges, such as the JIFT framework. Expand to include PMI. • Near term areas for collaboration include optimization of 5D problems, scalable parallel 2D FFTs, and analysis techniques for big data

7. Breakout Charge Questions • What technical breakthroughs in science and engineering research can be enabled by exascale platforms and are attractive targets for Japan-US collaboration over the next 10 years? • Please prioritize discussions around opportunities for collaborative Japan-US R&D relationships. See slide 4: • ITER operations supported by first principles based, multiphysics burning plasma simulations • including core and edge plasmas, including L-mode to H-mode transition • Routine application of 5D gyrokinetic code(s) to the whole volume ITER plasma, including some 6D domains • Whole device modeling, including ITER relevant physics such as energetic particles • Some physics modules may still need to be provided externally: e.g., RF wave modules • RF modeling for full 3D ITER device scale, full-f PIC for RF • Predictive simulation of disruption avoidance and mitigation • Good qualitative, and semi-quantitative understanding of plasma-material interactions

8. Breakout Charge Questions, continued • What is the representative suite of applications in your research area, available today, which should form the basis of your co-design communication with computer architects? • How are these applications currently constrained by compute and data resources, programming models, or available software tools? • What are the gaps in available applications and application workflows, and requirements to fill these gaps? • Which of these are ripe for collaboration within the context of Japan-US cooperation? • Gyrokinetics, MHD, RF, IPS (framework) • Constraints include compute time resources, network topology, on memory data management, scalable libraries, programming models for FORTRAN, memory bandwidth

9. Breakout Charge Questions, continued • How can the application research community, represented by a topical breakout at this workshop, constructively engage the vendor community in co-design? • How should these various aspects of the application and architecture be optimized for effective utilization of exascale compute and data resources? • Consider all aspects of exascale application: formulation and basic algorithms, programming models & environments, data analysis and management, hardware characteristics.

10. Breakout Charge Questions, continued • How can you best manage the “conversations” with computer designers/architects around co-design such that (1) they are practical for computer design, and (2) the results are correctly interpreted within both communities? • What are the useful performance benchmarks from the perspective of your domain? • Are mini-apps an appropriate and/or feasible approach to capture your needs for communication to the computer designers? • Are there examples of important full applications that are an essential basis for communication with computer designers? • Can these be simplified into skeleton apps or mini-apps to simplify and streamline the co-design conversation • The development of mini-apps to enable a variety of experimental investigations without the burden of large complex code systems seems reasonable. But there is concern about the sustainability and capturing the true physics of the problems. • XGC has an older version that has been used in this way.

11. Breakout Charge Questions, continued • Describe the most important programming models and environment in use today within your community and characterize these as sustainable or unsustainable. • Do you have appropriate methods and models to expose application parallelism in a high-performance, portable manner? • Are best practices in software engineering often or seldom applied? • Going forward, what are the critically important programming languages? • On which libraries and/or domain-specific languages (DSL) is your research community dependent? • Are new libraries or DSL’s needed in your research domain? • Are these aspects of your programming environment sustainable or are new models needed to ensure their availability into the future? • Appropriate methods and models exist, but best practices are seldom applied • Fortran, c, c++, python • mpi, openMP, openACC, adios, petsc, scalapack, dakota, fftw

12. Breakout Charge Questions, continued • Does your community have mature workflow tools that are implemented within leadership computing environments to assist with program composition, execution, analysis, and archival of results? If no, what are your needs and is their opportunity for value added? • For example, do you need support for real-time, interactive workflows to enable integration with real-time data flows? • We use adios for some codes. The IPS is also used for a set of codes.

13. Breakout Charge Questions, continued • What are the new programming models, environments and tools that need to be developed to achieve our science goals with sustainable application software?

14. Breakout Charge Questions, continued • Is there a history, a track record in your research community for co-design for HPC systems in the installed machines in the past, and is there any co-design study done for these systems to document the effectiveness of co-design? • There is a history of co-design, both in the US and Japan, where existing codes have benefitted from expertise making them highly scalable on existing machines (K machine, Titan, etc.) • There has been no systematic study of the effectiveness of co-design within fusion energy sciences. Some individual examples have data showing improvements.