Overview of the Flash Center

1. Overview of the Flash Center Don Lamb Annual Site Review October 28-29, 2004 (http://flash.uchicago.edu)

2. The Ultimate Goal: Understanding Astrophysical Flashes X-ray burst: GS 1826-24 Type Ia SN: SN 1994D Picture credits: NASA Cosmic nucleosynthesis: how is the stuff we’re made of produced? Guideposts to the universe: using exploding stars as “rulers” for the universe Nova Cygni 1992

3. Our “Overarching Goals” • To build a new generation code for computing astrophysical thermonuclear flashes involving compact stars • Incorporate advances from CS, applied math, physics • Verify and validate code and its components • To produce great science • Astrophysics and physics • Computational sciences [CS, Applied math, Math] • Training/education

4. Flash Code Highlights During Past Year • Updated Code Project Plan • Primary objectives for FY04 are to • Release Flash 2.4 – Done • Support addition of modules to Flash 2 required for astro simulations of Type Ia SNe – Done • Develop and implement Flash 3 Framework – Done • Implement a basic set of units in Flash 3 – Done • Run first application of Flash 3 (“end-to-end” test) – Done • Identified and helped to solve a series of problems w. ASC platforms (e.g., ALC and QSC)

5. More Flash Code Highlights • Organized and hosted Flash Tutorial • Researched, selected, purchased, and installed “ellipse” Linux cluster: • Sought and received invaluable advice from CS group at ANL • Cluster has 16 nodes (32 processors) • Includes 13 Tbyte RAID system • Processors and graphics cards selected are identical or similar to those used at ANL • Cluster will be used primarily for code development and visualization

6. Flash Code Future Objectives • Primary objectives for FY05 are to • Implement remaining units in Flash 3 • Create Flash 3 Developer’s Guide • Explore open design issues (I/O, gravity, particles) • Interact successfully w. • Astro group to support, document, maintain, and bug-track code; and to solve platform problems • Comp Phys group to ensure framework support for new algorithms • Visualization group to bring Flashview into Flash 2 and 3

7. Flash CS and Visualization Highlights During Past Year • Flash selected as Marquee application for LLNL BlueGene/L • Flash 3 is now running as “benchmark application” on Blue Gene/L • Flash 3 is driving developments in CS and Viz • Steadily increasing importance of visualization for astrophysics simulations has led to creation of Visualization group w. Mike Papka as leader

8. Validation Highlight: 3-D Simulations of Shock-Cylinder Experiment • Basic morphology of our 2-D simulations agrees with experimental data and simulations of others, but details differ across all groups • With Todd Dupont (basic physics group), we have done extensive study to determine initial conditions consistent with experimental data • We have developed quantitative, relevant, practical metrics • Our 3-D simulations demonstrate that flow field is fully 3-D in nature • Experimenters are planning to make measure- ments in vertical plane, prompted by our simulations Weirs et al. (2004)

9. Flash Comp Phys and Validation Future Objectives Primary objectives during FY05 are to: • Continue development and testing of low-Mach number, implicit hydro solver • Continue development and testing of level-set flame-tracking module • Improve flame module, taking into account results of “Astrophysical Flame Laboratory” simulations • Continue effort to build local radiation transfer capability for Type Ia SN light curves and spectra • Publish results of validation studies of LANL shock cylinder experiments • Initiate further validation studies w. LANL and LLNL

10. Flash Astrophysics Highlights During Past Year • Updated Astrophysics project plan • Continued work on novae (Plewa, Rubini, Pan, Truran) • Won INCITE award of 2.7 million cpu-hours on Seaborg for Flash simulations of Type Ia SNe • Carried out studies of sub-grid flame model using “Computational Astrophysical Flame Laboratory” (Messer) • Carried out whole-star simulations of Type Ia SN deflagration phase (Calder) • Octant models of central ignition • Whole-Star models of central ignition • Whole-Star models of off-center ignition • Discovered possible new Type Ia SN mechanism (Plewa)

11. Discovery of New Type Ia SN Mechanism • Whole-star 3-D simulations show that slightly off-center ignition creates hot bubble of material that rises rapidly to surface of white dwarf • As bubble breaks through surface, hot material flows across surface of star at high velocity, pushing material in surface layers of star ahead of it • Flow converges at opposite point on surface of star; resulting compression raises temperature and density of surface material, initiating detonation • First model of Type Ia SN in which detonation occurs “naturally” (i.e., without being put in by hand) • Star pre-expands while hot bubble material is flowing across surface of star, so that when detonation occurs, density of star has decreased enough that both nickel and intermediate mass elements are produced

12. Answers to Key Questions Re Type Ia SNe In Gravitationally Confined Detonation Model • How does insensitivity to initial conditions happen? —off-center ignition accommodates some variations in initial conditions, but more study definitely needed • What produces the range of Ni masses that are observed in Type Ia SNe that may be the origin of the Phillips relation? —variation in initial conditionsof off-center ignition produces range in masses of hot bubble, range in amounts of pre-expansion, and therefore range in amounts of Ni, but more study definitely needed • How does a detonation occur in a medium that has no walls (i.e., that is gravitationally confined)? —gravitationally confined detonation (colliding shocks in focused flow of hot material across white dwarf surface), but more study definitely needed • Why do 1-D models do better than 2-D models and much better than 3-D models? —bubble burns only a few percent of MWD,producing a pre-expanded star that is nearly spherically symmetric when detonation occurs; i.e., mimics 1-D model!

13. Criteria for Success • “Achieving scientific leadership in the chosen application discipline:” • Have achieved scientific leadership in nova problem • Are now a “player” in Type Ia supernova problem • “Demonstration of scientific breakthroughs and/or solutions to important problems through advanced computational science:” • Demonstrated viable solution to previously unsolved problem of pre-nova mixing • Discovered promising new Type Ia supernova mechanism (“Gravitationally Confined Detonation”)

14. Flash Center-Wide Team Effort on Type Ia SN Problem • Pre-2003 – Flash Code developed • Pre-2003-present – Flash Code maintained • 2003 – Flame capturing module developed and implemented • 2003-present – Flash Code migrated to new ASC platforms (QSC, ALC, BlueGene/L,…) and problems solved • 2003 – 3-D entire star slightly off-center simulations done

15. Off-Center Entire Star Simulation of Deflagration Phase of Type Ia SNe

16. Flash Center-Wide Team Effort on Type Ia SN Problem • Pre-2003 – Flash Code developed • Pre-2003-present – Flash Code maintained • 2003 – Flame capturing module developed and implemented • 2003-present – Flash Code migrated to new ASC platforms (QSC, ALC, BlueGene/L,…) and problems solved • 2003 – 3-D entire star slightly off-center simulations done • 2003-2004 – Multi-pole gravity developed and implemented • 2003-2004 – Multi-grid gravity speeded up

17. Flash Center-Wide Team Effort on Type Ia SN Problem • 2004 – Tracer particles developed and implemented to determine nucleosynthetic yields • 2004 – Distributed nuclear burning network being developed in order to follow detonation phase • 2004 – Data transfer rate from ASC platforms improved • 2004 – Cluster purchased, installed, and operating system problems solved • 2003-present – Flashview and visualization of Type Ia SN deflagration phase simulations done (e.g., movies) • 2004 – Type Ia SN post-deflagration phase 2-D simulations done

18. 2-D Simulation of Post-Deflagration Phase of Type Ia SNe

19. Flash Center-Wide Team Effort on Type Ia SN Problem • 2004 – Tracer particles developed and implemented to determine nucleosynthetic yields • 2004 – Distributed nuclear burning network being developed in order to follow detonation phase • 2004 – Data transfer rate from ASC platforms improved • 2004 – Cluster purchased, installed, and operating system problems solved • 2003-present – Flashview and visualization of Type Ia SN deflagration phase simulations done (e.g., movies) • 2004 – Type Ia SN post-deflagration phase 2-D simulations done • 2004 – Discovery of possible new Type Ia SN mechanism

20. Flash Center-Wide Team Effort on Type Ia SN Problem • 2003-present – Boussinesq reactive flow simulations done, leading to new insights • 2003-present – “Computational Astrophysical Flame Laboratory” simulations done, leading to new insights and improved sub-grid flame model

21. Self-Regulating Turbulent Flame Steady-state turbulent flame speed does not depend on small-scale physics: Sl=1.07 x 106 cm/s Sl=2.14 x 106 cm/s S = 1.41 x 1013 cm2 S = 2.84 x 1013 cm2 Messer et al. (2004)

22. Flash Center-Wide Team Effort on Type Ia SN Problem • 2003-present – Boussinesq reactive flow simulations done, leading to new insights • 2003-present – “Computational Astrophysical Flame Laboratory” simulations done, leading to new insights and improved sub-grid flame model • 2004 – New diagnostics developed for “Computational Astrophysical Flame Laboratory” • 2004 – First Monte Carlo radiation transfer code calculations of spectra for Flash Type Ia SN hydro simulations

23. First Contact Between Flash Hydro Simulations of Type Ia SNe and Observation Kasen and Plewa (2004)

24. Flash Center-Wide Team Effort on Type Ia SN Problem • 2003-present – Boussinesq reactive flow simulations done, leading to new insights • 2003-present – “Computational Astrophysical Flame Laboratory” simulations done, leading to new insights and improved sub-grid flame model • 2004 – New diagnostics developed for “Computational Astrophysical Flame Laboratory” • 2004 – First Monte Carlo radiation transfer code calculations of spectra for Flash Type Ia SN hydro simulations • 2004-present – Development and testing of low-Mach number hydro solver (will allow us to investigate “smoldering phase”) • 2004-present – Development of level-set interface capturing module

25. Future Flash Astrophysics Objectives Primary scientific objectives during FY05 will be to carry out large-scale, integrated, multi-physics simulations of Type Ia SNe • Focus on deflagration phase • Carry out verification studies • Resolution studies of center and off-center ignition • Carry out studiesof sensitivity to initial conditions • Parameter study of initial offset distances and velocities • Study effect of turbulence • Carry out 2-D simulations of detonation phase • Study nucleosynthetic yield, using tracer particles • Construct spectra for some Flash models

26. Program of Enrico Fermi Institute Mini-Symposium on Type Ia SNe

27. Reference Frame Column in Physics Today by Leo Kadanoff

28. Visibility of Flash Center in Scientific Community and Beyond • 2004-present – Award of INCITE time (2.7M cpu-hours) on Seaborg and selection of Flash code as Marquee application on BlueGene/L • Article on “Coming Crisis in Large-Scale Simulations” featuring Flash Center results will appear in a future Physics Today (Doug Post) • Educational and public outreach • Major upgrade of Flash Center website (incl. menu of Flash movies) • Showing of Flash Center movies at Adler Planetarium “Short Course” for planetarium staff and directors and disseminated to many planetaria • Development of 3-D movies of wind-wave mixing in novae and deflagration phase (“jellyfish”) of Type Ia SNe • Creation of exhibit of Flash Center results and movies for Sci-Tech, Hands-On Science Museum in Aurora, IL • Invitation to provide multi-projector Flash Center movies for exhibition at Smithsonian Air and Space Museum in Washington, DC, and for dissemination to planetaria throughout U.S.

29. Visibility of Flash Center in Scientific Community – Refereed Papers • On Variations in the Peak Luminosity of Type Ia Supernovae (Timmes, Brown, and Truran, ApJ, 590, 83, 2003) • Response of Model and Astrophysical Flames to Curvature and Stretch (Dursi, et al., ApJ, 595, 955, 2003) • Linear Instability of Astrophysical Flames in Magnetic Fields (Dursi, ApJ, 606, 1039, 2004) • Morphology of Rising Hydrodynamic and magneto-Hydrodynamic Bubbles from Numerical Simulations (Robinson, et al. ApJ, 601, 621, 2004) • On Heavy Element Enrichment in Classical Novae (Alexakis, et al., ApJ, 602, 931, 2004) • A Comparison of High-Resolution 3D Numerical Simulations of Turbulent Rayleigh-Taylor Instability: Alpha-Group Collabpration (Dimonte, et al., Phys. Fluids, 16, 1668, 2004) • Validating Astrophysical Simulation Codes (Calder, et al. CiSE 6, 10, 2004) • On the Nonlinear Evolution of Wind-Driven Gravity Waves (Alexakis, et al. Phys. Fluids, 16, 3256, 2004) • Type Ia Supernova Explosion: Gravitationally Confined Detonation (Plewa, Calder, and Lamb, ApJ, 612, L37, 2004) • Laboratory Astrophysics Experiments for Simulation Code Validation: A Case Study (Calder, Ap&SS, in press, 2004) • Type Ia Supernovae: Simulations and Nucleosynthesis (Brown, et al. Nuclear Physics A, in press, 2004) • Type Ia Supernovae: An Asymmetric Deflagration Model (Calder et al., ApJ, submitted, 2004)

30. Visibility of Flash Center in Scientific Community – Talks • 2 invited talks at APS/DPP in Albuquerque in October 2003 (Dwarkadas, Rosner) • 3 invited talks at Chicago Workshop on Type Ia SNe in November 2003 (Khoklov, Calder, Brown) • Talk at LES Workshop at Caltech in December 2003 (Plewa) • 2 oral talks at 192nd AAS Meeting in Atlanta, GA, in January 2004 (Alexakis, Calder) • 4 invited talks at 5th HEDLA Meeting in Tucson, AZ, in March 2004 (Rosner, Khokhlov, Calder, Lamb) • 3 invited talks at Ringberg Workshop on Nuclear Astrophysics in March 2004 (Calder, Brown, Truran) • Invited talk at Workshop on Combustion and Type Ia SNe in Sedona, AZ, in May 2004 (Plewa) • 2 talks at ASC Alliance/Laboratory Verification and Validation Workshop, La Jolla, CA, in July 2004 (Weirs, Plewa) • Talk at 9th International Workshop on Physics of Compressible Turbulent Mixing, Cambridge, UK, in July 2004 (Weirs) • 3 invited talks at Conference on AMR Simulations in Cosmology, Durham, UK, in June 2004 (Dubey, Gallagher, Sheeler)

31. Visibility of Flash Center in Scientific Community – Talks (Cont.) • 2 invited talks at INT Workshop on Supernova Theory and Nucleosynthesis in Seattle in July 2004 (Lamb, Truran) • 4 invited talks at INT Workshop on Type Ia SNe in Seattle in August 2004 (Bronson, Calder, Lamb, Truran) • Astronomy Department seminar at Notre Dame in September 2004 (Calder) • Astronomy Department colloquium at Illinois in September 2004 (Plewa) • Physics Department seminar at Illinois in September 2004 (Plewa) • Astronomy Department colloquium at Maryland in September 2004 (Plewa) • Astronomy Department colloquium at Vanderbilt in November 2004 (Calder) • Astrophysics seminar at Fermilab in Winter 2005 (Plewa) • Invited talk and contributed talk at Texas Symposium on Relativistic Astrophysics in Palo Alto, CA, December 2004 (Khokhlov, Messer) • 4 contributed talks at AAS meeting in San Diego, CA, January 2005 (Brown, Messer, Truran, Zhang)

32. Flash Center Visitor Program • Marko Arienti (Caltech) visited for two weeks in February; he worked w. Alexei Khoklov on simulations of detonations in solids • Dave Dearborn (LLNL) visited in April; he gave a Flash Center colloquium on Earth-crossing asteroids • Peter Hoeflich (U. Texas at Austin) visited for two weeks in April and again in September; he worked w. Alexei Khokhlov to port radiation transport modules to Chicago for eventual use in simulating Type Ia SNe light curves and spectra, and participated in the EFI Mini-Symposium on Type Ia SNe • Snejana Abarzhi (Stanford Turbulence Center) visited in early May; she gave a talk on the Rayleigh-Taylor instability • Wolfgang Hillebrandt (Max Planck Institute fur Astrophysik), who is on the Flash Center External Advisory Committee, visited in early May; he gave a Cosmology Seminar on Type Ia SNe • Luigi Iapichino (Max Planck Institute fur Astrophysik), visited for a week in May; he worked w. people on the smoldering phase of Type Ia SNe

33. Flash Center Visitor Program (Cont.) • Inma Dominquez (U. Granada, Spain) is visiting for a month; she is learning to use the Flash Code and is working w. members of the Center on models of the progenitors of Type Ia SNe • Paul Woodward (U. Minnesota) visited in late May; he is interested in contributing his interface tracking algorithm to Flash; his group has great experience w. flow diagnostics from which we can learn a great deal; visit was very productive for both sides • Francesco Rubini (Italy) visited for six months and will return early next year to continue working on the smoldering (pre-outburst) phase of novae • Dan Kasen (Johns Hopkins) visited for one month this summer; he will be worked w. Tomek Plewa to do first radiation transfer calculations of spectra for new Type Ia SN mechanism • Marco Comparato (Italy) will visit in November and December • Yingjie Liu (Georgia Tech) will visit in August 2005 to work on level set methods for interface tracking

34. Flash Center Folks Are Joining the Labs • Lori Freitag (cs group) permanent staff member, LLNL • Scott Wunch (basic phys group) research fellow, SNL (Livermore) • Alex Heger (astro group) research fellow, T Division, LANL • Andrew Siegel (code group) permanent staff member, MCS Division, ANL • Katherine Riley (code group) staff member MCS Division, ANL • Frank Timmes (astro group) National Security Fellow and • T-6 Group Leader, LANL • Andrei Draganescu (basic phys group) staff member, • SNL (Albuquerque) • Greg Weirs (comp phys & validation group) staff member, SNL (Albuquerque)

35. Flash Center Folks Are Going To Top-Notch Positions • Ed Brown (astro group)Assistant Professor, Department • of Physics and Astronomy, Michigan State University • Jonathan Dursi (astro group) Postdoctoral Fellow, CITA, Toronto, Canada • Andrea Mignone (astro group) postdoc in Torino, Italy • Alexandros Alexakis (astro group) postdoc in NCAR Advanced Study Program • Annie Sandman (solar MHD) did senior thesis w. Flash Center and graduated spring 2004, is now graduate student at Rice University

36. Recruitment in Past Year • New senior faculty memberin A&A who is actively working on Type Ia SN problem (Alexei Khokhlov) • Visiting senior faculty member in math (Henri Berestycki) • Two new postdoctoral research associates in astro group from pool of ~ 40 applicants(Alexei Poludnenko and Ju Zhang) • Two new postdoctoral research associates in comp phys and V&V group from pool of ~ 80 applicants(Dahai Yu) • Three new staff members in code group (Katie Antypas, Jignesh Joshi, Noel Taylor) • New system administrator (Shawn Needham) • Two new graduate students to work w. Flash (first-year students Justin Johnsen in A&A and Andriy Zhyglo in Physics • Undergraduate student to work w. Flash (Jason Sherwin)

37. Education • CS and computational science courses being taught: • A&A, CS and Math Department courses (scientific computation, numerical hydro, CFD, computational astrophysics, visualization) • CS Master’s Program courses (intro. to programming, parallel programming, adv. software engineering, Java, adv. Java) • CS Master’s Program has been a vital source for recruiting code group members (Gallagher, Needham, Sheeler) • Computer Science Department degree program in computational mathematicsis in its first year and growing: • Program involves courses in CS, Math, and other sciences • Flash Center expects to support 2 students in this program in 2005-2006 • PhD degrees awarded and expected: • Alexakis, Draganescu, Dursi, and Mignone graduated in 2004 • Winn expected to graduate in June 2005

38. Vision for Future of Flash Center Beyond the Next Three Years Supernova Acceleration Probe (SNAP) scheduled for 2013-2020 Flash Center Type Ia SN hydro and radiation transfer simulations

39. Type Ia SNe – An Overarching Theme of Flash Center • 1997 – Flash Center proposed and funded • 1998 – Observations of Type Ia SNe led to discovery that rate of expansion of universe is accelerating • 1997-present – Flash Code developed • 2003 – Flash Center begins work on Type Ia SNe • 2003 – DOE and NASA commit to Joint Dark Energy Mission (SNAP) • 2004 – Flash Center discovers new Type Ia SN mechanism • 2004 – First Flash Center spectra of Type Ia SNe • 2005-2013 – Flash Center will be able to contribute to design of SNAP instruments and to scientific observing strategy • 2013-2020 – SNAP mission • 2013-2020 – Flash Center will be able to contribute to • understanding Type Ia SNe through comparison of simulations and observations • minimizing systematic errors that will limit accuracy with which properties of Dark Energy can be determined

40. Specifics of Flash Center Beyond the Next Three Years • Code and CS • Core support from University of Chicago • Major additional support from ANL (CS and Viz) • Flash code continues to be a benchmark application for world’s fastest computers and driver for advances in CS and Viz • Comp Phys and Validation • Seek support through collaborations with NNSA labs to do validation experiments (e.g., at NIF) • Strong effort this year to renew collaboration w. Bruce Remington and others at LLNL • New postdoctoral research associate has been hired to increase activity in this area • Astrophysics • Create “Type Ia SN center” at University of Chicago • Support and participate in SNAP mission (e.g., in studies of dark energy and cosmology)

41. Responses to 2003 Review Team Report RecommendationAction Strategic Direction: • Develop 5-year plan for integrated simulations w. milestones Done • Maintain strategic focus on Type Ia SN problem Done • Prioritize implicit solvers, transport, and validation Done Flash as a Community Code: • Develop and release explicit design documentation for Flash 3.0 Done • Create a Developer’s Guide Underway • Ensure that new physics packages and code modules are easily implemented Done • Visualization should be a routine feature available to Flash users Underway • Flash Center should host annual workshops or conferences for Flash user/developer community Done • Flash design and implementation plan needs to accommodate production usage of Flash as well as continuous development Done

42. More Responses to 2003 Review Team Report RecommendationAction Verification and Validation: • Rigorously pursue VerificationUnderway • Establish a V&V methodology that is an integral part of the astrophysics research and Flash development Underway General: • Keep effort to develop Flash 3.0 going Done • Take advantage of existing numerical methods and techniques wherever possible Done • Enhance mentoring of staff and students Underway • Improve internal mechanisms for collaboration and communication, esp. wrt. graduate students Underway • Strengthen visitor program Done • Increase recruitment of graduate students Underway • Develop “curriculum” for Flash graduate students, including formal courses as well as training and seminars Underway • More and improved office space for staff and post docs Underway

43. Issues and Challenges • Code group needs increased manpower and resources to • Maintain Flash code • Support large-scale astro simulations • Make deep changes to Flash code & research challenging code issues • Comp phys group needs increased manpower and resources to • Support new astro simulations • Simulate experiments at NNSA labs – an activity vital to future of Flash Center beyond the next three years • Astrophysics group needs increased manpower and resources to exploit ability Flash code now has to address key astrophysics problems • Major funding issue: Alliance Centers funding uncertain at a time when we need to • Support diversified code effort • Expand validation effort • Exploit Flash code for key astrophysics problems

44. ASC Platform Usage

45. Issues and Challenges • Access to computational resources is a major issue: • As was already true at the time of the 2003 site review, we can use every cycle we can lay our hands on to do highly significant astrophysical simulations (i.e., we are now “cycle-bound” in the science that we are able to do). • We have aggressively sought to exploit ASC platforms to the full and have sought access to non-ASC platforms (e.g., INCITE proposal to use Seaborg). • We are aggressively seeking access to new ASC platforms (e.g., Blue Gene/L), but these machines will not come on line for a year or more. • ALC was designated as the main platform for Alliance centers in November 2003; however, it is an experimental machine. Close collaboration w. the ASAP CRT and w. consultants at LLNL Computing have been instrumental in our beginning to use ALC, but no production run has been successfully carried out to date. Recent progress encourages us to believe that we will be able to make full use of ALC in the near future. • The problems w. the ASC platforms temporarily placed a large burden on the code group, delaying the start of work on Flash3 until February 2004. • This year we had a harder time exploiting ASC computational resources. We are therefore appreciative that Frost will continue to be available to the Alliance centers, and are working with LLNL Computing to achieve reliability on ALC for Flash runs.

46. … which brings us to Questions and Discusson