Japan Atomic Energy Agency (JAEA) CCSE, Center for Computational Science & Engineering

1. JAEA’s HPC Analytic Challenge!Large-Scale Numerical Analysis of Nuclear Power Plant in Distributed Computational Environment Japan Atomic Energy Agency(JAEA) CCSE, Center forComputational Science & Engineering

2. JAEA: the union of JAERI & JNC for a new research organization for nuclear energy JFY2005 • JAEA: New Organization(Independent Administrative Institution) Employee: 4,400 Budget: ~JY200B (based on request) • Institute to be organized for full-scope R&D from basic research to development toward commercialization of Nuclear Energy • CCSE: Center for Computational Science Systems JFY２００４ • JAERIEmployee: 2,186 • Budget*: JY91B • Basic Research of Atomic Energy, Nuclear Energy System, Neutron Science, Nuclear Safety, Radiation Utilization, Fusion … • JNCEmployee: 2,259 • Budget*: JY114B • R&D for Nuclear Energy Untilization (FBR, spent-Fuel Reprocessing, Fuel Fabrication, Rad-Waste Treatment & Disposal …) * Budget from Government

3. CCSE Ueno, Tokyo

4. NPP is a facility to acquire energy through heat produced by nuclear fission. Mission of CCSE Contribute to safe operation and maintenance of Nuclear Power Plants (NPPs) underrealistic seismicevents by computational science & engineering Since… Seismic experiments of real-world NPPs are impossible

5. Real-world NPP Energy is exchanged and transferred among parts Energy loss will lead to a change in response of entire NPP Previous Simulation approach The parts were unified as a single structure, and relationship among parts had been neglected Challenge Interactions and interdependencies of parts must be considered to analyze full-scale NPPs Background

6. Challenge 1 Propose ‘Assembled-Structure Analysis’

7. Construct integrated system on ITBL grid to execute large-scale simulation Simulation procedures, from selection of datasets to visualization were made seamless on ITBL grid Why Grid? • Our future target • Storage : 5 Tera bytes (Mesh, Result data) • Memory usage : 200 Tera bytes (Matrices) Challenge 2 Too large to store on one super computer

8. Assembled-Structure Analysis • Status Quo • Elastostatic analysis on ITBL-Grid • SGI altix, IBM pSerise690, NEC Sx-6 and Fujitsu PRIMEPOWER • Parallel and distributed visualization technology • AVS/ITBL • Energy transfer among parts are considered • Deformation at part boundaries are consistent • Memory usage : 0.5 Tera byte(36milion DOFs)

9. Successfullysimulated NPP involving over 35 million DOFs as assembled structure. HPC analytics challenge “Honorable Mention” in just-closed SC|05 Conclusion • In order to contribute to safe operation and maintenance of NPPs, new approach called assembled-structure analysis was proposed • Integrated system for “assembled structure analysis” was developed on the ITBL grid

10. Interactions of inner components, buildings, multi-physical models, etc Entire Nuclear Power Plant simulation under realistic seismic events Future works Highly accurate interdependencies and interaction among parts

11. Thank you!

12. 1PE 2PE 3PE 4PE ITBLFEM –Penalty method and Shape Function- Part-1 Part-2 Part-3 Part-4 ITBLFEM-ASSEMBLE 1PE 2PE 3PE 4PE Connection element By using the connectivity information among parts, assembly-structure is constructed. On the parts boundaries, penalty constraints taking into account the shape functions are implemented.

13. Parallel Finite element analysis code 1PE 1PE Inner nodes 1PE 2PE Outer nodes 1PE 2PE Outer nodes Inner nodes : coupling components 2PE 2PE nodes The data structure of the ITBLFEM is node-based with overlapping elements. The stiffness matrix construction, and matrix-vector and vector inner product computations in the iterative solvers are local processes. Each domain passes necessary data at the overlaps in the iterative solver phase.

14. Test simulation <3-parts> <displacement>

15. DNS of bubbles in a sound field Radially oscillating bubbles experience an attractive or repulsive force, resulting sometimes in coalescence This is an elementary physical step observed in cavitation bubble clouds

16. Mechanism of the interaction force Strong attractive force acts when bubbles are larger than equilibrium sizes Weak repulsive force acts when bubbles are smaller than equilibrium sizes Thus, time-averaged force is attractive

17. Overview Various physical phenomena are as a result of interactions with other phenomena Coupled simulations are necessary to really understand phenomena Issues • Concurrency in distributed software • High-Performance Computing GRID can provide both Objectives • Development of competent coupler on ITBL GRID • Application of developed coupler to FSI simulations Fluid-Structure Interaction

18. ITBLFEM Code B Code A Partition n Partition n Partition 1 Partition 1 MPI MPI Coupler Fluid-Structure Interaction ITBL network 2. Spawn coupler & simulation software 3. Carry out coupled simulation 1. Choose arbitrary simulation software Communication + Interpolation

19. Fluid-Structure Interaction FSI simulation of finite-length elastic pillar Two square pillars Single circular pillar

20. Overview 2007: J-PARC (Japan Proton Accelerator Research Complex) Use of liquid mercury asneutron sourceby MW-classproton beam(nuclear spallation) Mercury target vessel: SUS316 liquid mercury circulates inside it proton beam neutrons Nuclear Spallation Mercury Target Problem Damage to the vessel shell due tocavitation (micropits)

21. Objective Carry out coupled simulation of liquid mercury and metal vessel, and assess integrity of vessel Mesh of fluid (BFC) Mesh of solid (shell element) Displacement distribution of vessel in each axial direction Nuclear Spallation Mercury Target

22. Calculation environment Tokai Laboratory in JAEA ITBLFEM GUI ③ TME Altix3900 Solver ① STARPC PC Terminal ② STAMPI ITBL Internet pSeries690 SX-6 Necessary condition Solver Solver ① Parallel computation between different model computers ② Communication by way of the Internet. ③ Job control on distributed computational environment Ueno Center for Computational Science and Systems in JAEA

23. ① STARPC Secure and Seamless Communication Internet ① STARPC • User is identified by using the X.509 certificate. • The mechanism of single sign-on is installed. User Execution computer Intranet A Authentication system Execution computer Firewall ITBL Server Intranet B Firewall Execution computer Encapsulation by HTTPS Communication mechanism ITBL Server Starpc is a communication mechanism of ITBL middleware. It enables the communication beyond firewall with security kept. Execution computer

24. ② STAMPI Internet User application User application User application stampi stampi stampi ② STAMPI Parallel computation using heterogeneous supercomputer cluster Intranet A Intranet B Execution computer Execution computer Execution computer Feature STAMPI is a MPI-based communication library which enables parallel computation using heterogeneous supercomputer cluster. (conformation to 18 models) User can achieve parallel calculation among different models of computers by linking of libraries of STAMPI. Interface of MPI1.2 standard

25. ③ TME User application User application User application stampi stampi stampi Internet ③ TME Job control on distributed environment Feature TME is a distributed program execution tool. Programs and data are shown as icons. Connecting icons as data flow diagram, workflow of an application can be made. Using TME, users can easily construct and execute their applications. Input Output Execution computer Execution computer Execution computer

26. ITBL-UNICORE Interoperability Trial SX-8 HLRS Tokai Laboratory in JAEA Solver Altix3900 Solver Internet Solver Solver Aim Job submission from ITBL client to UNICORE computers, in vice versa. More computers are shared by making ITBL cooperate with UNICORE, and a large-scale calculation is enabled. pSeries690 SX-6 Ueno CCSE in JAEA

27. Achievement method • A “Connection server” is introduced, and ITBL and UNICORE are mediated. • Function(From ITBL to UNICORE) • Conversion ofITBL’s request into AJO. • Submission of AJO request into UNICORE site. • Mutual forwarding of In/Output file. • Monitoring execution process • Function (From UNICORE to ITBL) • Recognize ITBL site as a virtual Usite. • Conversion of AJO into ITBL’s request • Communication with ITBL • Monitoring execution process

28. System Configuration Client GUI (applet, html) ITBL→UNICORE Request UNICORE Client GUI UNICORE→ITBL Starpc I/F AJO Connection Server ITBL Site Usite RMI skeleton RMI stab Server Certification filter Job Relay Servlet Starpc Control Servlet AJO UNICORE Colaboration Interface Gateway RDB Starpc I/F UNICORE Colaboration Module Unicore Cert Controler Front Server NJS AJO DB AJO is generated from ITBL’s request Starpc Relay Process ITBL→UNICORE UUDB IDB UNICORE→ITBL Starpc I/F Gateway AJO Manager Proxy Server TSI NJS AJO is converted into ITBL’s request. TSI Starpc I/F BSS ITBL Colaboration Application Tool A Tool B user program RDB Super-com(Vsite) user program Super-com ITBL Cert

29. Use Scene1（ITBL → UNICORE） Use Scene1（ITBL → UNICORE） • On ITBL’s TME（Task Mapping Editor）, UNICORE’s Computers’（Vsite’s） Program (Executable software) can be registered. GUI on TME (Job Configuration Definition) Push “Run All” to execute Job Output File ※On ITBL Computer Program On UNICORE Computer （Vsite） Input Files ※on ITBL Computers

30. Use Scene2（UNICORE → ITBL） • UNICORE recognize ITBL as Usite and Computers on ITBL as Vsite.Script on Vsite is transferred to ITBL’s Computers and Executed. Description example of Script task UNICORE Client GUI Import task(Input) ※ Files on Vsite USite corresponding to ITBL Site Script task (script, command) Export task(Output) ※ Files on Vsite VSite corresponding to ITBL computer

31. ITBL Middleware CCSE/JAEA

32. ITBL as an example of Grid Implementation ITBL as an example of Grid Implementation ITBL is one of ‘e-Japan National Priority Program’ • ‘e-Japan National Priority Program’ aims • To be the most advanced IT country in 2005 • To carry out 103 programs ITBL budgeted $160M for five years (2001-2005) • Schedule for R&D and Deployment • 2001-2002:Development, Implementation & Testing • 2003-2005:Deployment with Evaluation & Verification • 523 Users • 39 Organizations • 31 Computers & 17softwares • (2005.July) • ITBL aims to establish virtual laboratory until 2005 • Virtual Laboratory in Japanese Science & Engineering • Organization by, for, to the Interest people • Organization exists on World Area Network

33. Super SINET Node(10Gbps) SINET Node(1Gbps) User ITBL-Network ITBL Site • - 13 sites • 560 users • (70 organizations) • - 6 communities The main computer in ITBL-Network (18 types, 26 computers, 45.3TFlops) SGI Altix3900 Fujitsu PrimePower IBM pSeries690 Hitachi SR11000 NEC SX-6 Compaq SC/EX40 NEC SX-7 Hitachi SR8000 ・・・・・・・・・・・・ CRAY SV1EX Current Status of ITBL Network • 13 sites, 560 users (70 organizations)and 6 communities • 18 types, 26 computers (45.3TFlops in total)

34. C, C++, Java Application ITBL library Procedure RPC stub Web Browser Procedure ITBL Web GUI Firewall Computer Starpc User’s application HTTPS GridX tool Stampi ITBL Server ___________ ___________ ___________ ………. Stampi Computer X.509 certificate X Window base Application User’s application ITBL Site User’s Terminal Usage of ITBL Middleware • Web GUI • C, C++ and Java library • Secure communication tools (Grid-X, etc.)

35. Front Server Data Server computer LAN HTTPS computer Internet Relay Server FW User terminal ITBL Site Communication Mechanism • Starpc (Seamless-Thinking-Aid Remote Procedure Call) • Designed to compatible with firewalls • Authentication mechanism using X.509 certificate • Secure communication using HTTPS protocol

36. MPI Program MPI Program MPI Communication Environment • Stampi (Seamless-Thinking-Aid Message Passing Interface) • Conformation to MPI1.2 standard • Supports dynamic process creation and parallel I/O specified in MPI2.0 • Runs on 21 types of computer

37. Program Execution Environment • TME (Task Mapping Editor) • Graphical workflow definition and execution • Automatic program execution and data staging mechanism • Execution of workflow script using ITBL libraries

38. Slave Node Master Node PE1 PE2 PE3 PE4 p read field p isosurface struct PE1 PE2 PE3 PE4 PE1 PE2 PE3 PE4 High Performance Visuzlization • Parallel Support Toolkit(PST) forAVS/Express • developed to accelarate visulaization and manipulating large-scale data using parallel and distributed processing • data decomposition, distribution, large dataset handling, data streaming, and asynchronous computation

39. Remote Visualization • 2D and 3D viewer to display remote data 2D Viewer 3D Viewer

40. Video window Meeting controler Whiteboard Research Community Support Tools • Video Meeting tools • Common data folder • BBS