Introduction to FLASH 2.x

1. Introduction to FLASH 2.x Developers: A. Bodoh-Creed, A. Caceres, A. Calder, L. J. Dursi, W. Freiss, B. A. Fryxell, T. Linde, K. M. Olson, P. M. Ricker, K. Riley, R. Rosner, D. Sheeler, A. Siegel, F. X. Timmes, N. Vladimirova, G. Weirs, K. Young, M. Zingale

2. Outline • Current Status • FLASH architecture • PARAMESH / Data structures • Test Suite • Scaling / Performance

3. Current Status • FLASH 2.1 is the current released version • written in a mixture of Fortran 90, C, and python • support routines written in perl, Java, shell, and IDL. • Modules can be written in either Fortran 90 or C — API is provided for both languages • code is ~ 280,000 lines • MPI is used for interprocessor communication • HDF/HDF5 are the primary output formats (parallel I/O is supported with HDF5). • Nightly test suite is run on 8 platform/compiler combinations.

4. Simulations that FLASH must handle • Wide range of compressibility • Wide range of length and time scales • Many interacting physical processes • Large variety of problems • Scaling to 1000s of processors • Many people in collaboration

5. Adaptive Mesh Refinement • Required because of the very large range of length scales • Benefits: • Resolution only where needed • Memory/compute time decrease • Can do bigger problems • Disadvantages • Code is much more complex • Loss of data locality • Frequent redistribution/load balancing • Irregular/unpredictable memory/message patterns • Refinement/derefinement is an art

6. Current Status (cont.) Flash contains: • Hydrodynamics: • PPM • Kurganov • Magnetohydrodynamics (compressible TVD scheme) • Adaptive Mesh Refinement • Material Properties • EOS (gamma-law, electron/positron degenerancy,...) • Conductivies, Viscosity, Species Diffusion • Multifluid database provides access to material properties (atomic weights, proton number, ...)

7. Current Status (cont.) • I/O • HDF • HDF5 (parallel and serial) • Source terms • Thermonuclear burning • Stirring • Gravity • Constant • Plane parallel • Poisson (multigrid and multipole) • Particles (active and passive) • Automatic generation of C wrappers to dBase calls

8. Under development • New mesh modules • PARAMESH 3.0 • Supports cell, edge, and face-centered and corner data • Allows for permanent or temporary guardcell storage • Uniform Grid • Standardized interfaces to modules (CCA) • New solvers (relativistic PPM, radiation transport, Hall MHD, anelastic) • Automated FLASH debugger (based on AMS) • FLASH user interface over Globus

9. FLASH Architecture • Each FLASH module is divided into 3 components • Configuration layer describes: • module dependencies • Default sub-modules • Runtime parameter definitions • Interface layer • provides a public API to the underlying FLASH data-structures • accessor/mutator methods to manipulate the data • all modules (including the mesh) exchange data through the interface layer. • Algorithm • single block / single processor routine • written in any language

10. FLASH Architecture • Each module is in its own directory • Sub-module directories contain code, Makefiles, and configuration files specific to a particular variant of a module. • Each sub-module inherits all the code, Makefiles, and Configfiles of the parent. • Source files at a given level in the directory hierarchy override files of the same name at higher levels. • All modules communicate with each other only through the interface layer, NOT common blocks.

11. Interface layer • The data-structures from the mesh are not directly available to a module writer, but instead through database call. Assuming a module provides a Config file with the line: VARIABLE dens The variable dens can be accessed via Real dimension(nxb, nyb, nzb) :: density lnblocks = dBaseGetLocalBlockCount() do block_no = 1, lnblocks call dBaseGetData("dens", block_no, density) call foo(density) enddo • Similar calls exist of the runtime parameters, physical constants, and scratch space data.

12. Setup script

13. Creating a New Problem • New problems are added by creating a problem directory under setups/ • A Config file in setups/problem-name lists any module requirements, i. e. REQUIRES materials/eos/helmholtz • An init_block function provides a routine that initializes the unknowns for a single block. • setup problem-name -auto [-1d, -3d] setups the problem, linking all of the modules you requested into the build directory, and creates the Makefile • A list of the valid runtime parameters for the current setup is generated in paramFile.{txt,html}

14. PARAMESH Tree Structure • The computational domain is broken into blocks • Each block has the same logical structure. • Each block is surrounded by a perimeter of guardcells to exchange information with neighboring blocks • The hierarchy of blocks is maintained by a 2d -tree structure. • A block is refined by bisection in each coordinate direction • Each level of refinement is a factor of 2 different from adjacent levels

15. Load Balancing • A Morton space-filling curve is passed through all the blocks in the tree to generate a 1-dimensional ordering. • Other space filling curves were tried w/ no real improvement • Space filling curve preserves locality and minimizes the surface/volume of the sub-domains • Work weightings are assigned to each block • Equal amounts of work are assigned to each processor • Refinement occurs every 4 timesteps for PPM

16. Flux Conservation • At jumps in refinement, a flux conservation step must be performed to ensure global conservation. • The fluxes from the finer mesh are taken as the more accurate fluxes. • The fine cell fluxes are added and replace the coarse cell flux in the neighboring zone. • The summing is area weighted to take into account geometry.

17. FLASH Test Suite • Each night, the current development version of FLASH is checked out of CVS. • flash_test submits each job in the suite (currently 26) on several different platforms. • Currently there are comparison tests, compilation tests, and performance tests. • Results are posted online nightly

18. FLASH Test Suite (cont.) • The output is compared against stored benchmarks to machine precision • Any differences are flagged as failures • New tests are added frequently

19. Visualization Extensive plot/analysis routines in IDL for analyzing FLASH data

20. Development Dynamics • ~10 developers make ~500 changes each month (each change involves 1 or more files). • Keeping the code in CVS is an absolute must. • CVS basically forces you to comment about your changes, so every change to the code is documented. • Each night, the minor version number of FLASH is incremented. • This version number is supplied by a function flash_release() that is automatically created by the Makefile at compile time. • Every output file indicates the exact version number of the code used to make the executable, as well as the machine, directory, and setup call used. • Combined with CVS, this makes it possible to go back to the exact source used for a particular simulation weeks/months/years after the calculation. • Maintaining accurate/up-to-date documentation is important for bringing new developers into the project.

21. Development Dynamics • A nightly GNU-style ChangeLog is produced from the cvs logs giving details of the recent changes. • Nightly testing is probably the most important part of the development cycle. • Code is documented using RoboDoc. • Scripts are used to enforce our coding standards and report on violations. • Code is periodically parsed by syntax checkers/lints like forcheck. • Performance/scaling tests are periodically run to make sure that architectural changes have not come at the expense of performance.