Inner Workings And Architecture of FLASH

1. Inner Workings And Architecture of FLASH Anshu Dubey & Katie Antypas June 4, 2006

2. Overview • What is FLASH • Basics of FLASH Architecture • Similarities and differences between versions 2 and 3 • Units • Inheritance • Naming • Basics behind a problem setup • Enough of an idea of these concepts to be able to look at a sample setup and understand what is happening.

3. The Flash Code Shortly: Relativistic accretion onto NS Flame-vortex interactions Compressed turbulence Type Ia Supernova The Flash code Parallel, adaptive-mesh simulation code Designed for compressible reactive flows Has a modern CS-influenced architecture Can solve a broad range of (astro)physics problems Portable- runs on many massively-parallel systems Scales and performs well Is available on the web: http://flash.uchicago.edu Gravitational collapse/Jeans instability Wave breaking on white dwarfs Intracluster interactions Laser-driven shock instabilities Nova outbursts on white dwarfs Rayleigh-Taylor instability Orzag/Tang MHD vortex Helium burning on neutron stars Cellular detonation Magnetic Rayleigh-Taylor Richtmyer-Meshkov instability

4. Physics Hydrodynamics PPM MHD Relativistic PPM Equation of State Ideal gas Multigamma Helmholtz Nuclear Physics Gravity Cosmology Particles Infrastructure Setup Mesh Management AMR: Paramesh UG: Parallel I/O hdf5, pnetcdf Monitoring Runtime and post-processing visualization Regular testing toolkit Unit test framework What FLASH Provides

5. A Little FLASH History BAM • FLASH0 • Paramesh2, Prometheus and EOS/Burn • FLASH1 • Smoothing out the smash • First form of module architecture & inheritance • FLASH2 • Untangle modules from each other (Grid) • dBase • Concept of levels of users • FLASH3 • Stricter interface control & unit architecture • Taming the database • Better encapsulation and isolation • Community supported and developed code

6. FLASH Audiences FLASH Application Developer End User Application Programmer • Works on just about everything • Grid development • Data access • Architecture • Initial conditions • Boundary conditions • Runtime parameters • Select units, and their specific implementations • Implement functions specific to an application, if they are not provided by FLASH • Develop physics Unit • Publish API • Use other unit’s API

7. FLASH Code Basics • An application code, composed of units/modules. Particular modules are set up together to run different physics problems. • Performance, Testing, Usability, Portability • Fortran, C, Python, … • More than 560,000* lines of code • 75% code, 25% comment • Very portable • Scaling to tens of thousand procs * Internal Release

8. FLASH on BG/L - Sod weak scaling Hydro Time: Constant Work Per Processor using AMR Total Time Number of Processors Source :K.M. Riley, Argonne National Lab

9. Hydro and Particles Scaling

10. Basic Computational Unit : Block • The grid is composed of blocks • Most often all blocks have same dimensions • Cover different fraction of the physical domain. • In AMR blocks at different levels of refinement have different grid spacing.

11. FLASH Architecture Four Cornerstones Setup tool assemble simulation Config files Tell setup how to Assemble simulation Unit Architecture API Inheritance Data management Driver Organize interactions Between units

12. What’s a FLASH Unit? • FLASH basic architecture unit • Component of the FLASH code providing a particular functionality • Different combinations of units are used for particular problem setups • Publishes a public interface for other units’ use. • Ex: Driver, Grid, Hydro, IO etc • Fake inheritance by use of directory structure • Interaction between units governed by the Driver • Not all units are included in all applications

13. FLASH Units Infrastructure Grid Runtime Params I/O monitoring Physics Profiling Hydro MHD Driver Gravity Burn Logfile Simulation

14. Unit Architecture Grid Data Module • Top level: • API • Unit Test block data time step etc Wrapper Driver Kernel

15. Unit Hierarchy Unit API/stubs UnitMain Common API implementation UnitSomething API implementation common More common impl Impl_1 Remaining API impl kernel Impl_2 Remaining API impl Impl_3 Remaining API impl kernel kernel kernel

16. Inheritance Through Directories: Eos Stub Implementations of the three functions at the top level Eos_ init Eos Eos_ wrapped There is only one subunit EosMain Eos/EosMain Replaces the stub with an implementation common to all formulations of EOS Eos_wrapped Gamma Specific implementation Eos/EosMain/Gamma implements gamma versions Of Eos_init and Eos Eos_init Eos Another implementation, which will have its own Eos and Eos_init etc. Multigamma

17. The Config File • Declare solution variables, fluxes • Declare runtime parameters • Sets defaults • Lists required, requested or exclusive units • Config files are additive down the directory tree - no replacements

18. FLASH Setup Script: Implements Architecture Python code links together needed physics and tools for a problem • Parses Config files to • Determine a self consistent set of units to include • If a unit has multiple implementations, finds out which implementation to include • get list of parameters from units • Determines solution data storage • Creates a file defining global constants set at build time • Builds infrastructure for mapping runtime parameters to constants as needed • Configures Makefiles properly

19. Pulling it All Together • Choose a problem simulation • Run setup to configure that problem • Everything is in a new top-level directory • ‘object’ • Make • Run • flash.par for runtime parameters • Defaults already set from particular units

20. Adding a new Simulation • A basic problem setup • Config file • Required physics modules • flash.par • Default list runtime parameter configuration • Simulation_initBlock • Initial conditions for the problem set block by block • Many other possible files: • Driver, Refinement algorithms, User defined boundary conditions • Any files in setup take precedence More about setting up your own problem in the hands on session later this afternoon

21. Provided Driver • Provided: Second order, state form, strang split FLASH3 FLASH2 flash.F90 Flash.F90 Initialize Loop over timesteps evolve adjust dt output visualize End loop Finalize Driver_initFlash Driver_evolveFlash Driver_finalizeFlash Driver Unit Driver_evolveFlash.F90 Loop over timesteps set time step call physics set time step repeat physics Grid_updateRefinement adjust dt output End loop evolve.F90 set time step do physics set time step repeat physics Mesh_updateGrid

22. Flash Grids : Paramesh • FLASH original design was Paramesh specific • Block Structured • All blocks have same dimensions • Blocks at different refinement levels have different grid spacings and thus cover different fractions of the physical domain • Fixed sized blocks specified at compile time • Every section of the code assumed knowledge of block size • Good for capturing shocks • Currently removing fixed block size requirement from code to open door for other mesh packages like a uniform grid, squishy grid and patched base grid In choosing Paramesh, the original FLASH code architects chose simplicity of the Paramesh structure over a patch based mesh.

23. Infrastructure : Mesh Packages in Flash Paramesh Uniform Grid • Block Structured • Fixed sized blocks • Specified at compile time • Not more than one level jump at fine coarse boundaries • one block per proc • No AMR related overhead

24. Building and Running an Application Runtime Inputs Driver Grid mesh Simulation/ setup I/O Physics Profiler

25. IO/Viz Outline • I/O Challenges with Scientific Computing • speed, portability, file size • serial, parallel multi-file, MPI-IO • I/O Libraries • hdf5 • parallel-netcdf • FLASH I/O specifics • parameter input file • checkpointing and restarts • plotfiles • Visualization • runtime visualization • fidlr3 • flashView

26. I/O Challenges: Portability Taking a basic binary output file from one machine and trying to use it on another often causes problems • Portability • big endian, little endian • size of types vary • data alignment in memory 4 byte int 2 byte int int int int double 24 0 4 8 12 16 20

27. I/O challenges: File size and Speed • File size • Files are huge --in FLASH a typical checkpoint file typically ~5 GB and up to 50 GB for a research simulation (plotfiles ~2GB) • Need access to large storage areas • Data transfer takes lots of time (~.5 MB/sec from labs) • I/O can be a major performance bottle neck • slow file system • writing to disk takes time It isn’t hard to have speed, portability or usability. It is hard to have speed, portability and usability in the same implementation

28. Serial I/O 0 1 2 3 4 5 processors • Each processor sends its data to the master who then writes the data to a file • Advantages • Don’t need a parallel file system • Simple • Disadvantages • Not scalable • Not Efficient File

29. Parallel I/O Multi-file 0 1 2 3 4 5 processors File File File File File File • Each processor writes its own data to a separate file • Advantages • Fast! • Disadvantages • can quickly accumulate many files • hard to manage • requires post processing

30. Parallel I/O Single-file 1 2 3 4 5 0 processors File • Each processor writes its own data to the same file using MPI-IO mapping • Advantages • single file • scalable (to how many procs is debatable...) • Disadvantages • requires MPI-IO mapping or other higher level libraries

31. 3 5 2 9 2 4 3 1 9 8 2 4 Parallel I/O single file 0 1 2 3 4 5 processors array of data Each processor writes to a section of a data array. Each must know its offset from the beginning of the array and the number of elements to write

32. I/O Libraries • FLASH works with 2 different I/O libraries • HDF5 • Parallel-NetCDF • Use MPI-IO mappings • Both Portable libraries • Both can translate data between systems • Scientific Data mostly stored in multidimensional arrays • FLASH3 also supports a basic direct FORTRAN I/O -- use only as a last resort!

33. FLASH I/O • FLASH can use either HDF5 or Parallel-NetCDF for I/O • Initially very optimistic about PnetCDF because it appeared to be ~ twice as fast as HDF5 Source :J.B. Gallagher on ASC White

34. Parameter Input File - flash.par • To change parameters in a simulation edit the flash.par file • Defaults are set in Config files and any value in the flash.par overides the Config values • Parameter MUST be declared in Config file!!! • Controls parameters for all units (examples below) • Grid parameters - lrefine_max, lrefine_min, geometry • Physics parameters - flame_speed • I/O parameters - restart, checkpointFileNumber

35. Flash Checkpoint (restart) Files • Checkpoint file -- a file that saves all the data you need to restart your simulation from the exact state it left off • Grid Data • refinement level • nodetype (parent, child) • bound box • coordinates • unknown variables (density, pressure, temperature etc) • Simulation Data • time • dt • timestep • MetaData • time stamp • build directory • setup problem etc For any number of reasons your simulation could be interrupted - the machine crashes, the queue window closes, the machine runs out of disk space etc. Rather than starting again from the beginning of your simulation you can restart your problem using a flash checkpoint files

36. Checkpoint Parameters • There are a number of ways to control the occurrence and spacing of checkpoint dumps (names have been clarified for FLASH3) • timesteps between dumps - (parameter: checkpointFileIntervalStep) • simulation time seconds between dumps - (parameter: checkpointFileIntervalTime) • wall clock seconds between dumps - (parameter: wall_clock_checkpoint) • very useful when running in a queue or when only a certain amount of time is available on a machine • creating a file called .dump_checkpoint will force a checkpoint file to be dumped immediately • creating a .dump_restart file will force a checkpoint and halt the code • rolling checkpoint ability - (parameter: rolling_checkpoint) • checkpoint files are big! 5 - 50GB for a large science run • this feature allows you to keep only a specified number of checkpoints to save disk space The FLASH code support website gives detailed explanations of all runtime parameters

37. Restarting a Simulation To restart a simulation, in your flash.par file set *the runtime parameter ‘restart’ = .true. *the checkpointFileNumber parameter to the desired checkpoint number • Because the checkpoint files save the exact state of the simulation you can restart your simulation from any FLASH checkpoint file • Can start with Parallel IO if previous run was written in serial • Can restart with a different number of processors than the previous run

38. 3 3 5 5 2 2 9 9 2 2 4 4 3 3 1 1 9 9 8 8 2 2 4 4 Restarting on a different number of processors 0 1 2 3 4 5 processors array of data restart 0 1 2 processors array of data

39. Flash Plotfiles and Particle Files Plotfiles and particle files output data for visualization and analysis purposes • Plotfiles are smaller than checkpoints • Only output variables needed for visualization or analysis • Data stored in single precision • Control plotfile frequency with the runtime parameters • plotfileNumber • plotfileIntervalStep • plotfileIntervalTime • Particle files stored in single precision • Control particle file frequency with the runtime parameters • particleFileNumber • particleFileIntervalStep • particleFileIntervalTime

40. Visualization

41. Visualization • Runtime Visualization • Fidlr3 • FlashView

42. Runtime Visualization • Runtime Visualization module • not automatically included -- can add it into your Modules file to set up problem with Runtime Viz • Allows user a quick and dirty view of the simulation • Only requires the widely available png library • Creates png files in current directory during runtime

43. Fidlr3 (FLASH IDL Routines) Fidlr is an application written in idl to visualize Flash checkpoint and plotfiles • IDL tool for 2d visualization • Available with FLASH directly • Can step through plots to view sequences or see movies • Can visualize 3d data by looking at slices • Make histograms • Additionally, if you have access to idl and want to do more analysis you can use the routines to read in data, statistical analysis, max min vaules

44. Example of 3d slice taken by FILDR3

45. FlashView FlashView is a 3d visualization package developed for FLASH data at Argonne National Lab by Mike Papka and Randy Hudson • Features • Reads native FLASH plotfiles • XYZ cut planes features so entire range of data for a variable can be projected on a plane • Renders 3d iso-surfaces of Flash Data • Fully parallel (both reading the file and calculations) • Parallelism allows for the ability to look at very large data sets interactively • Displays mulitple (up to 5) variables at the same time • various color maps, zoom, rotation, translation etc • shows grid structure • movies making • Currently only supports hdf5 format * Yes ... we think it is difficult to install too... * Key is to install exact version of recommended packages * Ask Dan if you have questions

46. FlashView

47. Example of an isosurface (single variable)

48. Another example with two variables

49. Another example/ isosurface with cut plane

50. Isosurface with Grid