How to create your application: The magic of the setup script

1. How to create your application:The magic of the setup script Lynn Reid May 23, 2007

2. Building and Running an Application Runtime Inputs Driver Grid mesh Simulation/ setup I/O Physics Profiler

3. The setup Script The primary jobs of the setup script are to: • Traverse the FLASH source tree and link necessary files for a given application to the object directory . • Find the target Makefile.h for a given machine. • Generate the Makefile to make the FLASH executable. • Determine solution data storage list and create Flash.h • Generate files needed to add runtime parameters to a given simulation. • Generate files needed to parse the runtime parameter file.

4. Simple setup hostname:Flash3> ./setup MySimulation -auto setup script will automatically generate the object directory based on the MySimulation problem you specify Sample Units File INCLUDE Driver/DriverMain/TimeDep INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/headers INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/mpi_source INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/source INCLUDE Grid/localAPI INCLUDE IO/IOMain/hdf5/serial/PM INCLUDE PhysicalConstants/PhysicalConstantsMain INCLUDE RuntimeParameters/RuntimeParametersMain INCLUDE Simulation/SimulationMain/Sedov INCLUDE flashUtilities/general INCLUDE physics/Eos/EosMain/Gamma INCLUDE physics/Hydro/HydroMain/split/PPM/PPMKernel INCLUDE physics/Hydro/HydroMain/utilities Try manually changing IO/IOMain/hdf5/serial/PM to IO/IOMain/hdf5/parallel/PM - Then run setup without the -auto flag If you don’t use the -auto flag, you must have a valid Units file in the object FLASH directory (FLASH3/object/Units)

5. setup Options: use ./setup -help

6. setup Shortcuts • The Flash3 setup script includes a “shortcuts” feature • Allows many setup options to be included with one keyword • To use a shortcut, add +shortcut to your setup line • Example: The shortcut ug is defined as: ug:--with-unit=Grid/GridMain/:Grid=UG: prompt> ./setup MySimulation -auto +ug this is equivalent to typing in unit options with -unit=Grid/GridMain/UG -unit=IO/IOMain/hdf5/serial/UG (because the appropriate IO is included by default) • Look in Flash3/bin/setup_shortcuts.txt for more examples and to define your own

7. Important Files Generated by setup

8. Additional Files created by setup • Flash.h contains • Problem dimensionality and size e.g. NDIM, MAXBLOCKS • Fixed block size dimensionality e.g. NXB, GRID_IJI_GC • Variable, species, flux, mass scalar numbers and list e.g. e.g. NSPECIES, DENS_VAR, EINT_FLUX • Possibly grid geometry GRID_GEOM • PPDEFINE variables showing which units are included e.g. FLASH_GRID_PARAMESH3 • Simulation_mapIntToStr.F90, Simulation_mapStrToInt.F90 • Converts text strings to equivalent index in Flash.h e.g. “dens” maps to DENS_VAR=1 • Similar functionality to FLASH2’s dbaseKeyNumber

9. Example Makefile.h Located in FLASH3/sites/<machine name> MPI_PATH = /usr/local/mpich-intelHDF5_PATH = /usr/local/hdf5-iccNCMPI_PATH = /usr/local/pnetcdf-icc FCOMP = ${MPI_PATH}/bin/mpif90CCOMP = ${MPI_PATH}/bin/mpiccCPPCOMP = ${MPI_PATH}/bin/mpiCCLINK = ${MPI_PATH}/bin/mpif90 FFLAGS_OPT = -c -r8 -i4 -O3 -real_size 64# -unroll -align -prefetch -pad -ip FFLAGS_DEBUG = -c -g -r8 -i4 -check bounds -check format -check output_conversion -warn all -real_size 64FFLAGS_TEST = -c -r8 -i4 -O2 -real_size 64CFLAGS_OPT = -c -O3 -D_LARGEFILE64_SOURCECFLAGS_DEBUG = -c -g -debug extended -D_LARGEFILE64_SOURCECFLAGS_TEST = -c -O2 -D_LARGEFILE64_SOURCECFLAGS_HDF5 = -I $(HDF5_PATH)/includeCFLAGS_NCMPI = -I $(NCMPI_PATH)/includeCFLAGS_MPI = -I$(MPI_PATH)/includeLFLAGS_OPT = -r8 -i4 -Vaxlib -lsvml -Ur -oLFLAGS_DEBUG = -r8 -i4 -Vaxlib -g -oLFLAGS_TEST = -r8 -i4 -Vaxlib -oLIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lzLIB_MPI = -L$(MPI_PATH)/lib -lfmpich -lmpichLIB_NCMPI = -L$(NCMPI_PATH)/lib -lpnetcdf

10. Adding a New Simulation • Create a directory with initial Uppercase letter in FLASH3/source/Simulation/SimulationMain • The Simulation directory has problem specific files. It needs: • A Config file describing the Units and runtime parameters needed by this problem • One or more flash.par files which each give a set of runtime parameter values for running the problem in different ways • Data modules for problem-specific parameters or storage • Simulation_data.F90 • Code specifying initial conditions • Simulation_init.F90, Simulation_initSpecies.F90, Simulation_initBlock.F90 • Makefile indicating necessary Fortran files • Problem specific code replacing default functionality in the main source tree of the code • IO_writeIntegralQuantities.F90, Grid_updateRefinement.F90

11. Config File • Needed in each Unit or Simulation directory • Declare solution variables, fluxes • Declare runtime parameters • Sets defaults and allowable ranges • Documentation • Lists required, requested, exclusive modules • Gives Unit-specific pre-processor symbols with PPDEFINE • Allows some implementations choices with LINKIF • Variables, Units are additive down the directory tree • Runtime parameters cannot be duplicated – • first one found wins! • Simulation directory Config file overrides main code • Provides warnings to prevent dumb mistakes

12. Basic Config File Syntax

13. More Config File Syntax

14. Config file example Required Units Alternate local IO routines Runtime parameters and documentation Additional scratch grid variable Enforce geometry or other conditions

15. Now what? Move on to programming the physics and geometry of your problem