Computational Cosmology Initiative

1. Computational Cosmology Initiative Report of the Task Force Scott Dodelson

2. Task Force • Scott Dodelson (Center for Particle Astrophysics), co-Chair • Don Petravick (Computing Division), co-Chair • James Amundson (Computing Division) • Jim Annis (Experimental Astrophysics) • Nick Gnedin (Theoretical Astrophysics) • Don Holmgren (Computing Division) • Jim Kowalkowski (Computing Division) • Andrey Kravtsov (University of Chicago) • Andreas Kronfeld (Particle Theory) Scott Dodelson

3. Task Force Charge from Hugh Montgomery, April 10, 2007 Develop, as necessary, and articulate the case for a strategic initiative for a computational cosmology computing project as part of the Fermilab computing capability. It will be important to: • Articulate an outstanding scientific case. • Discuss the existing and potential scientific collaboration across the US. Could this evolve into a national program? • put the proposed capability in the context of other computing capabilities such as that available from large national computing centers such as NERSC, leadership DOE, Terascale NSF centers. • put the proposed capability in the context of worldwide developments • discuss the technical aspects; this may include an R&D component. • discuss the infrastructure needs. • discuss the resource needs including the effort at different levels addressing not only the hardware but also the algorithmic aspects of a possible project. • discuss the connection to or synergies with any relevant other high performance computing or SciDACC efforts. • develop a business plan considering multiple funding sources. Prepare a written report considering two 3-year periods with the first period not thoroughly developed. This should take into account the funding cycle for opportunities such as SciDACC. The report should be limited to 20 pages maximum including all tables and figures using a reasonable font size. It would be interesting to receive a draft report by mid-June with the final report by July 31. Scott Dodelson

4. Task Force • Weekly Meetings • Briefings on: Lattice QCD (AK), Legal/Business Issues (DH), Computing Resources (Space, power, etc.) (DP), Science (AK,NG), Software Issues (AK,NG), Refining Charge (Mont) • Delivered First Draft in June 2007 • Sent for review to ~10 people; received responses from ~5; incorporated these in final draft • Final draft delivered August, 2007 Scott Dodelson

5. Science Case We have learned a lot from the cosmic microwave background because deviations from uniformity are small Measurements can be compared to theory which depends on parameters Scott Dodelson

6. Science Case Extracting information from observations of galaxies is much more difficult • Gravitational Instability is nonlinear • Baryons governed by hydrodynamics • Radiation Field affects and is affected by structure • Stars form, Supernovae explode, … Scott Dodelson

7. Science Case Akin to the difference between QED QCD Scott Dodelson

8. Science Case If we can obtain accurate predictions for a given set of cosmological parameters, the payoff will be enormous • Shape of the matter power spectrum depends on neutrino mass and inflation • Evolution of structure depends on dark energy • Gravitational wells determined by dark matter • Rich astrophysical questions: galaxy formation, reionization, black holes, quasars,… Scott Dodelson

9. Science Case Computing power – software and hardware – has improved so that the goal of obtaining accurate predictions for the distribution of matter in the universe is within reach Note: Focus on cosmological simulations from smooth initial conditions Scott Dodelson

10. Science Case Example: Weak Lensing. Leaving out physics of star formation leads to incorrect predictions in the “dark energy” region. Scott Dodelson

11. Science Case: Finding #1 “Cosmological simulations are powerful tools to understand the growth of structure in the universe. This understanding is interesting and important in its own right, but also is essential for extracting precision information about fundamental physics - such as dark energy and dark matter - from upcoming astronomical surveys.” Scott Dodelson

12. Local Expertise Nick Gnedin (FNAL) and Andrey Kravtsov (UChicago) are collaborators and two of the strongest numerical astrophysicists in the world. They have assembled excellent teams of students and postdocs. The two groups are extremely productive (25 papers/183 citations since 1/06). Scott Dodelson

13. Local Expertise Gnedin and Kravtsov developed Adaptive Refinement Tree (ART) code • Implementation of Adaptive Mesh Refinement method • Refines on a cell-by-cell basis • Full 4D adaptive • Includes - dark matter dynamics - gas dynamics and chemistry - radiative transfer - star formation, etc Scott Dodelson

14. Local Expertise Scott Dodelson

15. Local Expertise: Finding #2 “Groups at the University of Chicago and Fermilab have the tools in the form of state-of-the-art codes to contribute to this important area.” Scott Dodelson

16. Resources • As of today, Computing Division maintains an 8 core cluster for cosmology • Other U.S. groups have many more resources: • ITC, Harvard: 316 CPU Opteron, 264 CPU Athlon • LANL: 294 CPU Pentium4 (just for cosmology) • SLAC: 72 CPU SGI Altrix, 128 CPU Xeon cluster • Princeton: 188 CPU Xeon cluster • UWash: 64 CPU Pentium cluster • International Groups are far ahead • Virgo Consortium: 670 CPU SparcIII, 816 CPU Power4 • CITA: 270 CPU Xeon cluster, 2000 CPU dual Pentium (shared) Scott Dodelson

17. Resources In the two years since the inception of Distributed European Infrastructure for Supercomputing Applications, 8 large-scale allocations were made for computational cosmology projects at European supercomputer centers with an average allocation size of 1 million hours or more. For comparison, only 1 computational cosmology project of similar size was approved in the last three years through the DOE INCITE program. Scott Dodelson

18. Resources: Finding #3 “The local resources for these groups are currently inadequate to run the largest, highest resolution simulations. This inadequacy is representative of a problem on a national scale. In contrast, European groups have pooled resources and currently run the most powerful cosmological simulations.” Scott Dodelson

19. Infrastructure @ FNAL • Space/Power for Hardware not a problem • CD currently supports many machines (>20,000 cores) • Software development for LQCD/Physics for Accelerators • Build/manage releases for software for D0,CDF, CMS … • Experience benchmarking code; optimizing hardware • Data Handling • Project Management Scott Dodelson

20. Infrastructure @ FNAL: Finding #4 “Fermilab has the infrastructure to host a cosmological computing initiative on a national scale.” Scott Dodelson

21. Proposal What have we done so far? • Received FRA grant ($75k) • Received funds for hardware from Kavli Institute for Cosmological Physics at UChicago ($100k) • Contributed funds from Theoretical Astrophysics (Rocky money) ($60k) • Successfully obtained LDRD at Argonne for software development and ultimately joint postdocs • Expect to have ~500 cores @ FNAL by December 2007 Scott Dodelson

22. Proposal What are we asking for from FNAL in FY08? • Funds to continue building local cluster ($200k) • Support for cluster from CD (0.2 FTE) • Software Developer from CD (0.5 FTE) Scott Dodelson

23. Proposal Scott Dodelson

24. Proposal Long Term Goal Develop a national program in computational cosmology powerful enough to exploit fully Stage IV dark energy experiments Scott Dodelson

25. Back-up Slide 1: PAC Advice “Should opportunities for expanding the scope of Fermilab R&D activities emerge in the future (noble liquid detectors for Dark Matter searches and water Cerenkov detectors for long-baseline neutrino experiments are among the various possibilities), they should be evaluated in the context of the overall national scientific strategy and funding constraints, taking into account the following criteria: 1. Scientific and technical excellence; 2. The uniqueness and impact of the Fermilab contribution, given its existing infrastructure and technical expertise; and 3. The interest among Laboratory personnel in collaborating on the project. Ideally, Laboratory management would help external proponents to identify relevant personnel who might be available to participate.“ Scott Dodelson

26. Back-up Slide 2: Resources Scott Dodelson