1 / 31

Hands-On Flash

Hands-On Flash. Setting up Your Own Simulations. Outline. Setting up an out of the box simulation Customizing Flash for a new simulation (i.e., how to set up your problem) Simple visualization and analysis with idl using fidlr3.0. Setting up an out of the box problem.

anahid
Download Presentation

Hands-On Flash

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Hands-On Flash Setting up Your Own Simulations

  2. Outline • Setting up an out of the box simulation • Customizing Flash for a new simulation (i.e., how to set up your problem) • Simple visualization and analysis with idl using fidlr3.0

  3. Setting up an out of the box problem

  4. Flash Source to Executable • Flash comes as source code consisting of source code components known as units • Going from Flash source code to executable consists of • The setup step: specifying which source units to include in the application and linking together the source files • The compilation step: running make to compile the flash executable

  5. The Setup Script The primary job of the setup script is to: • Traverse the FLASH source tree and link necessary files for a given application to the object directory • Find the target Makefile.h for a given machine. • Generate the Makefile to make the FLASH executable. • Generate files needed to add runtime parameters to a given simulation. • Generate files needed to parse the runtime parameter file.

  6. Example Makefile.h Located in FLASH3/sites/<machine name> MPI_PATH = /usr/local/mpich-intelHDF5_PATH = /usr/local/hdf5-iccNCMPI_PATH = /usr/local/pnetcdf-icc FCOMP = ${MPI_PATH}/bin/mpif90CCOMP = ${MPI_PATH}/bin/mpiccCPPCOMP = ${MPI_PATH}/bin/mpiCCLINK = ${MPI_PATH}/bin/mpif90 FFLAGS_OPT = -c -r8 -i4 -O3 -real_size 64# -unroll -align -prefetch -pad -ip FFLAGS_DEBUG = -c -g -r8 -i4 -check bounds -check format -check output_conversion -warn all -real_size 64FFLAGS_TEST = -c -r8 -i4 -O2 -real_size 64CFLAGS_OPT = -c -O3 -D_LARGEFILE64_SOURCECFLAGS_DEBUG = -c -g -debug extended -D_LARGEFILE64_SOURCECFLAGS_TEST = -c -O2 -D_LARGEFILE64_SOURCECFLAGS_HDF5 = -I $(HDF5_PATH)/includeCFLAGS_NCMPI = -I $(NCMPI_PATH)/includeCFLAGS_MPI = -I$(MPI_PATH)/includeLFLAGS_OPT = -r8 -i4 -Vaxlib -lsvml -Ur -oLFLAGS_DEBUG = -r8 -i4 -Vaxlib -g -oLFLAGS_TEST = -r8 -i4 -Vaxlib -oLIB_HDF5 = -L $(HDF5_PATH)/lib -lhdf5 -lzLIB_MPI = -L$(MPI_PATH)/lib -lfmpich -lmpichLIB_NCMPI = -L$(NCMPI_PATH)/lib -lpnetcdf

  7. First Setup Example • From the main flash directory, FLASH3/, type ./setup Sedov -2d -auto command problem name dimension Setup script determines which units to include based on the problem name (rather than manually) To run setup you shouldn’t need to modify the setup script (FLASH3/bin/setup.py)

  8. Setup Options -auto setup script will automatically generate the Units file based on the problem you specify Sample Units File INCLUDE Driver/DriverMain/TimeDep INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/headers INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/mpi_source INCLUDE Grid/GridMain/paramesh/Paramesh3/PM3_package/source INCLUDE Grid/localAPI INCLUDE IO/IOMain/hdf5/serial/PM INCLUDE PhysicalConstants/PhysicalConstantsMain INCLUDE RuntimeParameters/RuntimeParametersMain INCLUDE Simulation/SimulationMain/Sedov INCLUDE flashUtilities/general INCLUDE physics/Eos/EosMain/Gamma INCLUDE physics/Hydro/HydroMain/split/PPM/PPMKernel INCLUDE physics/Hydro/HydroMain/utilities Try manually changing IO/IOMain/hdf5/serial/PM to IO/IOMain/hdf5/parallel/PM - Then run setup without the -auto flag If you don’t use the -auto flag, you must have a valid Units file in the top FLASH directory (FLASH3/Units)

  9. Setup Options • Another way to use a specific unit in setup is with the -unit option -unit=<unit> : force a specific unit to be used in setup ./setup Sedov -2d -auto -unit=IO/IOMain/hdf5/parallel/PM

  10. More Setup Options

  11. Even More Setup Options

  12. Setup Shortcuts • The Flash3 setup script includes a “shortcuts” feature • Allows many setup options to be included with one keyword • Defined in Flash3/bin/setup_shortcuts.txt • To use a shortcut, add +shortcut to your setup line • Example: The shortcut ug is defined as: ug:--with-unit=Grid/GridMain/:Grid=UG: ./setup Sod -auto +ug this is equivalent to typing in unit options with -unit=Grid/GridMain/UG -unit=IO/IOMain/hdf5/serial/UG (because the appropriate IO is included by default) • Look in Flash3/bin/setup_shortcuts.txt for more examples and to define your own

  13. setup Generated Files setup_call contains the options with which setup was called and the command line resulting after shortcut expansion setup_libraries contains the list of libraries and their arguments (if any) which was linked in to generate the executable setup_units contains the list of all units which were included in the current setup setup_defines contains a list of all pre-process symbols passed to the compiler invocation directly setup_flags contains the exact compiler and linker flags setup_params contains the list of runtime parameters defined in the Config files processed by setup setup_vars contains the list of variables, fluxes, species, particle properties, and mass scalars used in the current setup, together with their descriptions

  14. Customizing Flash for a new simulation (i.e., how to set up your problem)

  15. Adding a New Simulation • New Simulations are added by creating a directory in FLASH3/source/Simulation/SimulationMain • The Simulation directory gives people working on a particular problem a place to put problem specific code that replaces the default functionality in the main body of the code • It’s also a place to tell the setup script which units this problem will need from the rest of the code • In the Simulation directory, one needs to have: • A Config file to at least describe the units and runtime parameters needed by this problem • One or more flash.par files which each give a set of runtime parameter values for running the problem in different ways • Code specifying initial conditions and code for the simulation as a unit (e.g., initialization of the unit)

  16. Config File Syntax

  17. More Config File Syntax

  18. Even More Config File Syntax

  19. Required Code for a New Simulation • There are certain pieces of code that all simulations must implement: • Simulation_initBlock.F90: Sets initial conditions in a single block. • Simulation_init.F90: Reads the runtime parameters, and performs other necessary unit initializations. • Simulation_data.F90: A Fortran module which stores data and parameters specific to the Simulation. • Optionally, a sumulation could implement: • Simulation_initSpecies.F90: To give the properties of the species involved in a multispecies simulation

  20. Customized Code for a new Simulation • In a FLASH simulation directory, you can place code that overrides the functionality you would pick up from other code units • In the custom code you can modify: • Boundary conditions (Grid_applyBCEdge.F90) • Refinement criterion (Grid_markRefineDerefine.F90) • Diagnostic integrated quanties for output (in the flash.dat file), e.g., total mass (a default) or vorticity (IO_writeIntegralQuantities.F90) • Diagnostics to compute new grid scope variables (Grid_computeUserVars.F90) • In general, this is a place to hack the code in ways specific to your problem, and you can hack basically anything

  21. Example Non-trivial Setup: TwoGamma • TwoGamma is for a simple test of advecting two fluids having different gammas to investigate whether an instability develops at the interface between the two fluids. • This Simulation implements: • Simulation_initSpecies.F90, because it has multiple fluids • and Grid_applyBCEdge.F90, because it needs custom boundary conditions on the lower x edge of the domain

  22. TwoGamma Config File # configuration file for the TwoGamma target problem REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos/EosMain/Multigamma REQUIRES Multispecies REQUESTS IO # Parameters D sim_p0 constant pressure PARAMETER sim_p0 REAL 2.5e-0 D sim_rho1 density of the first fluid PARAMETER sim_rho1 REAL 1.0e-0 D sim_rho2 density of the second fluid PARAMETER sim_rho2 REAL 1.0e-0 D sim_cvelx initial velocity PARAMETER sim_cvelx REAL 0.1e-0 SPECIES FLD1 SPECIES FLD2

  23. TwoGamma flash.par # AMR parameters lrefine_max = 4 lrefine_min = 4 # simulation parameters basenm = "twogamma_" restart = .false. plotFileIntervalTime = 0.1 checkpointFileIntervalTime = 0.5 nend = 15000 tmax = 10.0 checkpointFileNumber = 0 plotFileNumber = 0 dtini = 1.e-10 dtmin = 1.e-10 cfl = .5 cvisc = .1

  24. TwoGamma flash.par (Cont.) smlrho = 1.e-10 smallt = 1.e-10 xmin = 0.0e0 xmax = 1.0 ymin = 0.0e0 ymax = 1.0e0 geometry = "cartesian" # variables for plotting plot_var_1 = "dens" plot_var_2 = "temp" plot_var_3 = "ener" plot_var_4 = "pres" plot_var_5 = "velx" plot_var_6 = "fld1" plot_var_7 = "fld2" xl_boundary_type = "user" xr_boundary_type = "outflow" yl_boundary_type = "periodic" yr_boundary_type = "periodic"

  25. TwoGamma Simulation_initSpecies.F90 subroutine Simulation_initSpecies() implicit none #include "Multispecies.h" #include "Flash.h" #include "Multispecies.h" #include "Multispecies_interface.h" call Multispecies_setProperty(FLD1_SPEC, A, 1.) call Multispecies_setProperty(FLD1_SPEC, Z, 1.) call Multispecies_setProperty(FLD1_SPEC, GAMMA, 1.66666666667e0) call Multispecies_setProperty(FLD2_SPEC, A, 4.0) call Multispecies_setProperty(FLD2_SPEC, Z, 2.0) call Multispecies_setProperty(FLD2_SPEC, GAMMA, 2.0) end subroutine Simulation_initSpecies

  26. TwoGamma Grid_applyBCEdge.F90 if(face==LOW) then select case (bcType) case(OUTFLOW) do i = 1,guard dataRow(i)= dataRow(guard+1) end do case(USER_DEFINED) select case(var) case(GAMC_VAR) dataRow(1:guard)=sim_gammac1 case(DENS_VAR) dataRow(1:guard)=sim_rho1 case(PRES_VAR) dataRow(1:guard)=sim_p0 case(VELX_VAR) dataRow(1:guard)=sim_cvelx case(VELY_VAR) dataRow(1:guard)=0.0 case(VELZ_VAR) dataRow(1:guard)=0.0 case(ENER_VAR) dataRow(1:guard)=max(0.5*(sim_cvelx**2)+sim_int1,sim_small)

  27. Simple visualization and analysis with idl using fidlr3.0

  28. Fidlr (Flash idl Routines) 3.0 • 1 and 2d visualization • 2d slices of 3d data • Has idl routines to read the HDF5 and NetCDF Flash checkpoint and plotfiles. • xflash3: Gui frontend • Can call load routines from idl command line and manipulate checkpoint data interactively • Lives in tools/fidlr3.0 in your source

  29. setenv XFLASH3_DIR flash-root-path/tools/fidlr3 setenv IDL_DIR idl-root-path setenv IDL_PATH ${XFLASH3_DIR}:${IDL_DIR}:${IDL_DIR}/lib Setting up and Running idl for fidlr • Setting up environment correctly very important • To get color, run idl with start_linux: • % idl start_linux • IDL> xflash3

  30. xflash3 – the fidlr GUI

  31. fidlr Interactive Examples Read in density from file, put it on a uniform grid, and make a contour plot: IDL> dens = loaddata(‘sedov_2d_6lev_chk_0001’, ‘dens’, XCOORDS = x, YCOORDS=y) IDL> contour, dens, x, y Read in a dataset, with full AMR details: IDL> read_amr, ‘sedov_2d_6lev_hdf5_chk_0001’, $ TREE=tree, PARAMETERS=params, DATA=data, STORED_VARS=vars

More Related