Production Version of Flash : 1.61

1. Production Version of Flash: 1.61 Paul Ricker University of Chicago The Flash Code Group A. Caceres, A. Calder, T. Dupont, J. Dursi, B. Fryxell, T. Linde, A. Mignone, K. Olson, P. Ricker, R. Rosner, K. Riley, A. Siegel, F. Timmes, H. Tufo, N. Vladimirova, G. Weirs, K. Young, M. Zingale

2. Overview • Flash 1.61 in context • Algorithmic architecture • Automated verification • Summary

3. Flash 1.61 in context

4. FLASH 1.6 Architecture, Optimization FLASH 2.0 Data FLASH 0.0 Initial FLASH 1.0 Organization Flash framework evolution

5. Physics modules (* - new this year) • Compressible hydrodynamics • Piecewise-parabolic method (PPM) • Weighted essentially non-oscillatory (wENO) * • Tadmor central-difference scheme * • Compressible magnetohydrodynamics * • Total variation diminishing (TVD) * • Equations of state (EOS) • Partially degenerate stellar EOS(Coulomb corrections) * • Mixture of perfect gases * • Source terms • Nuclear burning -variety of reaction networks * • Single-mode turbulent energy injection * Flash capabilities

6. Physics modules (continued) • Gravitational field • Externally imposed (constant, plane parallel *, point source *) • Self-gravity * • Multipole summation * • Multigrid * • Couchman FFT method * • Diffusion * • Thermal diffusion (explicit) * • Radiation (implicit) * • I/O formats • Hierarchical data format (HDF) version 4 and 5 * Flash capabilities

7. Implementation • 83,000 lines of FORTRAN 90 + C • ~ 24,000 in driver + mesh package • ~ 59,000 in physics modules • Block-structured adaptive mesh using PARAMESH • Parallelization using MPI • I/O using HDF4/5 • Ports • ASCI platforms (Red, Blue, Nirvana) • Cray T3E • Linux clusters (Intel & Alpha) Flash 1.61 implementation

8. Algorithmic architecture

9. Performance • Must be able to do the largest problem possible • Modularity • Must be easy to switch initial/boundary conditions • Must be easy to add/change solvers • Ease of adoption • Should work with existing F77 routines • Should not require many extra packages • Must be portable • Should use standard data formats • Ease of experimentation • Must be easy to vary problem parameters • Should permit "quick-look" visualization Framework design requirements

10. Driver steady runtime_parameters log_file performance_monitor physical_constants init_from_scratch() init_from_checkpoint() init_flash() init_block() solve() output() tree data structure time_dependent init_block() evolve() output() timestep() Diffusion Other Mesh Hydro Gravity Source_terms Materials init() fill_guardcells() test_refinement() refine_derefine() init() tstep() set_ext_guardcell() hydro_3d() init() tstep() set_ext_guardcell() gravity_3d() init() tstep() src_terms() eos_3d() eos() abund_renorm() query_iso_db() Paramesh Uniform Explicit Implicit constant burn Gamma point_mass iso13 Helmholtz ... Unsplit PPM Split Poisson ... cool heat PPM WENO PPM Multigrid Multipole Flash interfaces

11. Modules file Config files Plaintext files describing modules to include in code Plaintext files describing module interrelationships Hydro init() tstep() set_ext_guardcell() hydro_3d() Makefile INCLUDE PARAMETER REQUIRES DEFAULT EXCLUSIVE Explicit Implicit tstep() tstep() Setup script PPM Split Unsplit hydro_3d() etc. Makefile Configures build directory according to Modules file instructions for a specified machine and problem type hydro_3d() hydro_sweep() update_soln() hydro_1d() hydro_3d() PPM WENO PPM hydro_1d() etc. Makefile hydro_3d() etc. Makefile hydro_3d() etc. Makefile make F90 Implementation: compile-time "polymorphism"

12. Architectural consistency • "Object-oriented" structure • Mimic virtual functions • Cumulative inherited Makefiles • Only internal and mesh common blocks • Exclusive and non-exclusive modules • Test driver capability • Simulation services • Runtime parameter contexts • Physical constants database (plus unit conversions) • Materials properties database • Logging and timing libraries • New modules Changes from version 1.0

13. Automated verification

14. Flash calculations • Wide range of compressibility • Wide range of length and time scales • Many interacting physical processes • Only indirect validation possible • Rapidly evolving computing environment • Many people in collaboration Flame-vortex interactions Compressible turbulence Laser-driven shock instabilities Nova outbursts on white dwarfs Richtmyer-Meshkov instability Cellular detonations Helium burning on neutron stars Rayleigh-Taylor instability

15. Automated verification of Flash • Want verification to be easy and routine • Flash includes several test problems • Sod, Sedov, Woodward-Colella blast wave, advection, wind tunnel, Shu-Osher, isothermal collapse, Jeans instability, … • More are added as needed • Automated testing using flash_test • Compiles and runs Flash • Compares against benchmark results using focu • focu (Flash Output Comparison Utility) • Local or global error criteria • Generates a short report for flash_test

16. Stored benchmark data Test database Indexed by date; output files created by previous invocation of flash_test Runtime parameter files and configuration information for all test problems included with code Comparisons via focu Jobs tracked via lockfile mechanism Site database Queue and OS information for each configured site; job and job submission templates Output Log file indicating success of each compile/run/compare; output files Automated verification architecture flash_test

17. Summary • Flash... • Is a small, focused, but flexible framework • Offers state-of-the-art simulation capabilities • Is extensively verified • Achieves high performance (Fryxell talk) • 0.24 TFlop/s on ASCI Red (2000 Gordon Bell finalist) • Scales to > 6,000 processors • Is producing exciting science (Truran, Dupont talks) • Continues to evolve (Siegel talk)