1 / 28

The Flash Code

This talk provides an overview of the Flash code, its architecture, performance, and scaling. It also discusses the current and future development of the code with a focus on adaptive mesh refinement and physics modules. The Flash Code Group, led by Bruce Fryxell and Andrew Siegel, is responsible for the development, maintenance, and testing of the code.

hanes
Download Presentation

The Flash Code

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Flash Code Bruce Fryxell Leader, Code Group Year 3 Site Review Argonne National Laboratory Oct. 30, 2000

  2. Outline Talk 1 – Bruce Fryxell Overview of Flash Adaptive Mesh Refinement Performance and Scaling Year 3 Integrated Calculation Talk 2 – Paul Ricker Current production version of Flash Flash Code architecture Flash physics modules Code verification Talk 3 – Andrew Siegel Development version of Flash The future of Flash

  3. The Flash Code Group Bruce Fryxell Group Leader Andrew Siegel Code Architect Development, Maintenance, Testing Architecture Team Physics Modules Calder, Linde, Mignone, Olson, Ricker, Timmes, Tufo, Weirs, Zingale Calder, Dursi, Olson, Ricker, Timmes, Tufo, Zingale Caceres, Ricker, Riley, Vladimirova, Young

  4. Overview of Flash Driver Time Dependent Initialization Parallel I/O Steady Nuclear Burning Hydro EOS Gravity Diffusion Mesh

  5. Year Three Upgrades • Evolution to object-oriented code architecture • P. Ricker, A. Siegel talks • PARAMESH • PARAMESH 1 • “SHMEM” emulation replaced by native MPI • Unnecessary barriers removed • PARAMESH 2 (K. Olson poster) • Elimination of permanent guard cell storage • Capability to advance solution at all refinement levels instead of just at leaf blocks • Adaptivity in time • Guard filling in one direction at a time • New and upgraded physics modules • P. Ricker talk, many posters

  6. Other Accomplishments • Parallel I/O • HDF 5 • 10x improvement in I/O throughput • Documentation • Comprehensive user manual • http://flash.uchicago.edu/flashcode/doc • The physics and algorithms used in Flash • http://flash.uchicago.edu/flashcode/pubs • Code release • Friendly users – May 2000 • Astrophysics Community – Oct. 2000

  7. Adaptive Mesh Refinement • Reduces time to solution and improves accuracy by concentrating grid points in regions which require high resolution • PARAMESH (NASA / GSFC) • Block structured refinement (8 x 8 x 8 blocks) • User-defined refinement criterion – currently using second derivatives of density and pressure

  8. Flash / PARAMESH Block Guard Cells Interior Cells

  9. PARAMESH Tree Structure • Each block contains nd zones in d dimensions • Blocks stored in 2d-tree data structure • Factor of 2 refinement per level • Blocks assigned indices via space-filling curve 10 12 18 Refinement Level 6 11 21 20 14 19 10 9 8 7 2 18 6 11 12 14 2 16 17 15 13 19 20 21 7 8 9 5 4 17 13 15 16 1 3 4 5 3 1

  10. Load Balancing • Work weighted Morton space filling curve • Performance insensitive to choice of space filling curve • Refinement and redist-ribution of blocks every four time steps

  11. Example – X-ray Burst

  12. Performance Optimization • Single processor tuning • Reduction in number of square roots and divides • Loop fusion to eliminate unneeded arrays • Elimination of scratch arrays • Removal of unnecessary array copies and initializations • Replacement of string comparisons by integer comparisons • Use of vendor-supplied math libraries • Modification of often-used routines to permit in-lining on ASCI Red • Result • 90 Mflop/son 250 MHz R10000 (64 bit)

  13. Performance Optimization • Parallel optimization • Use of Jumpshot and to identify problem areas • Removal of unnecessary barriers • Packing of small messages in tree portion of code • Result • Good scaling to 1024+ processors • 238 Gflop/s on 6420 processors of ASCI Red for the year 3 integrated calculation • 2000 Gordon Bell prize finalist

  14. Scaling • Constant work per processor scaling • Shock tube simulation • Two-dimensional • Hydrodynamics, Adaptive Mesh Refinement, gamma-law equation of state • Relatively high communication to computation cost

  15. Scaling - Constant Work Per Processor Flash 1.6 – May 30, 2000

  16. Scaling - Constant Work Per Processor

  17. Scaling - Constant Work Per Processor

  18. Scaling • Fixed problem size scaling • Cellular detonation • Three-dimensional • Uses most of the major physics modules in the code • Relatively low communication to computation cost

  19. Scaling – Fixed Problem Size

  20. Scaling – Fixed Problem Size

  21. Summary of Scaling • As number of blocks per processor decreases, a larger fraction of the blocks must get their guard cell information from off processor • This causes deviation from ideal scaling when the number of blocks per processor drops too low • Of the three ASCI machines, this effect is most noticeable on Red, due to its relatively small memory per processor • Significant variation in timings on Nirvana between identical simulations

  22. Summary of Scaling • Significant improvement in cross-box scaling on Nirvana can be achieved by tuning MPI environment variables • Scalability on Blue Pacific is highly dependent on operating system revisions • Parallel efficiency for memory bound jobs • > 90% on Blue Pacific and Red • > 75% on Nirvana • Typical performance – 10-15% of peak on 1024 processors

  23. Integrated Calculation Cellular Detonation In A Type Ia Supernova See also: J. Truran talk F. Timmes poster Evening demos

  24. Why a Cellular Detonation? • Two of our target astrophysics calculations (X-ray bursts and Type Ia Supernovae) involve detonations • We can not resolve the structure of the detonation front in a calculation which contains the entire star • Want to do a study of a small portion of the detonation front to see if a subgrid model is necessary to compute • The detonation speed • The nucleosynthesis • This problem exercises most of the major modules in the code and thus serves as a good test of the overall code performance

  25. Integrated Calculation • 1000 processors on ASCI Blue Pacific • Effective grid size (if fully refined) • 256 x 256 x 5120 = 335 million grid points • Actual grid size • 6 million points at beginning of calculation • 45 million points at end of calculation • Savings from using AMR • 40-50x for first half of calculation • 7x at end of calculation • Total wall clock time ~ 70 hours

  26. Integrated Calculation • Generated 1.2 Tbyte of data • Half of wall clock time required for I/O • 0.2 Tbyte transferred to ANL by network for visualization • Used GridFTP to transfer files • 7 parallel streams to 7 separate disks • Throughput ~ 4 Mbytes/s • Total transfer time < 1 day

  27. 105 6 level 105 5 level 105 0 2 4 6 8 10 12 Simulation Time (10-8 s) Integrated Calculation

  28. Summary • Substantial progress made in Year 3 in improving and extending Flash • Flash is now being used to address many of our target astrophysics problems and is producing important scientific results • Flash achieves good performance on all three ASCI computers and scales to thousands of processors • Large 3D integrated calculation completed on ASCI Blue Pacific and data successfully transferred back to Chicago for analysis and visualization

More Related