Astronomy of the Next Decade: From Photons to Petabytes

1. Astronomy of the Next Decade:From Photons to Petabytes R. Chris Smith AURA Observatory in Chile CTIO/Gemini/SOAR/LSST

2. “Classical” Optical Astronomy • 1-4 investigators propose for telescope time • Obtain 1 to 5 nights • Travel to distant telescope site • Observe • or not: clouds (OUCH!) • Take 5 to 50 GB of data home (on tapes) • Reduce & Analyze “by hand” • Extract every detail from those bits • Often takes months per night of data Photons to Petabytes 2014

3. The “Modern Classic” Photons to Petabytes 2014

4. Optical Windows Photons to Petabytes 2014

5. Classical Astronomystill dominates new facilities • Even new large facilities (VLT, Gemini, ALMA, GMT, E-ELT) are and will be scheduled for “individual projects” • In units of nights, sometimes hours! • But methods are changing… • Sloan Digital Sky Survey led the way • Statistical analyses -> new discoveries • Surveys and science with massive datasets are growing, filling an important need Photons to Petabytes 2014

6. Today’s BIG Questions:Dark Energy & Dark Matter Dark Energy is the dominant constituent of the Universe. Dark Matter is next. 95% of the Universeis in Dark Energy and Dark Matter, for which we have little or no detailed understanding. 1998 and 2003 Science breakthroughs of the year, 2011 Nobel Prize Photons to Petabytes 2014

7. Attacking the Question of Dark Energy & Others • “Classical” approach won’t work • Not enough telescope time in 2-5 night “chunks” • LARGE SURVEYS • Goal: Provide large, uniform, well calibrated, controlled, and documented datasets to allow for advanced statistical analyses • Larger and broader collaborations provide both manpower and diverse expertise • NEED… • NEW INSTRUMENTS • NEW TELESCOPES • NEW METHODS Photons to Petabytes 2014

8. Sociology of Dark Energy • Dark Energy may be pushing the universe APART • But it is pulling the Astronomy, Physics, Mathematics, and CompScicommunities TOGETHER • New physics, new algorithms, new processing capabilities (h/w & s/w), new access methods (DBs, fast networks) Photons to Petabytes 2014

9. Selected Examples: Coming soon to nearby mountaintops… New Instruments (DECam) New Telescopes (LSST) Photons to Petabytes 2014

10. Dark Energy Survey (DES) • 5 year project to improve our understanding of Dark Energy • Key DOE/NSF collaboration: Fermilab/NOAO/NCSA • International collaboration: Brazil, UK, Spain, Germany • Characterize Dark Energy with four methods • Supernovae • Weak Lensing (also measure Dark Matter) • Galaxy clustering • Baryon Acoustic Oscillations • All depend on careful statistical analyses of large datasets Photons to Petabytes 2014

11. Dark Energy Camera • CAMERA: • 62 2048 x4096 pixel CCDs • 570 Megapixel camera • The largest focal plane for astronomy in S. Hemisphere Optical Lenses Photons to Petabytes 2014

12. DECam is here TODAY First light images: September 12, 2012 Fornax galaxy cluster Photons to Petabytes 2014

13. Photons to Petabytes 2014

14. Small Magellanic Cloud ~5 billion measurable stars in this single image! Photons to Petabytes 2014

15. A “modest” data challenge • Each image 1GB; 300–700 GB of raw data/night • Data must be moved from Chile to NCSA before next night begins (<18 hours), preferably in real time • YEAR 1: Each image transferred in <120 sec! • Data must be processed within <24 hours to inform next night’s observing: using NCSA resources • YEAR 1: Real-time pipeline processing on Tololo with LIneAQuickReducepipeline: robust and reliable • Initial processing completed at NCSA in <24 hours, still with only limited data quality specifications • TOTAL Dataset will be ~5 PB Photons to Petabytes 2014

16. The next step… ca. 1950 POSS (Photographic) ca. 2000 SDSS (Digital) ca. 2012 DES (Digital + Depth) ca. 2020 LSST (Digital Sky +Time Domain) Photons to Petabytes 2014

17. Next Step = LSST: Creating a “Digital Universe” • 8.4 M Telescope • 3.5 Degree Field Of View • Telescope Located in Chile on Cerro Pachón • 3.2 Billion Pixel Camera • ~40 Second Cadence • Two 15 second exposures • Full sky coverage every few nights • Advanced Data Management Systems • Public Data • Alerts of new events • Catalogs of object • Archives of images LSST is designed to image the whole sky every few nights for 10 years, giving us a movie-like window into our dynamic Universe. Photons to Petabytes 2014

18. The Large Synoptic Survey Telescope – Massively Parallel Astrophysics Survey the entire sky every 3-4 nights, to simultaneously detect and study: • Dark Matter via Weak gravitational lensing • Dark Energy via thousands of SNe per year • Potentially hazardous near earth asteroids • Tracers of the formation of the solar system • Fireworks in the heavens – GRBs, quasars… • Periodic and transient phenomena • ...…the unknown Photons to Petabytes 2014

19. 3.5 degrees (Full moon is 0.5 degrees) LSST 8.4 m Why is the LSST so unique? Primary Mirror Diameter Field of View 0.2 degrees Gemini South Telescope 8 m Photons to Petabytes 2014

21. Telescope and Site 30 m diameter dome 1.2 m diameter atmospheric telescope Control room and heat producing equipment (lower level) 1,380 m2 service and maintenance facility Base Facility 350 ton telescope Includes the facilities, and hardware to collect the light, control the survey, calibrate conditions, and support all LSST summit and base operations. Photons to Petabytes 2014

22. Camera Utility Trunk—houses support electronics and utilities Cryostat—contains focal plane & its electronics L3 Lens 1.65 m (5’-5”) Filter Focal plane L2 Lens L1 Lens Camera ¾ Section 3.2 Gigapixel science array – 10 square degree FOV! Wavefront and guide sensors 2 second readout 5 filters in camera Photons to Petabytes 2014

23. Petascale Data Management • Each image roughly 6.5GB • Cadence: ~1 image every 15s • 15 to 18 TB per night, 30TB “reduced”! • ALL must be transferred to NCSA archive center • within image timescale (17s), >>10 Gbps • REAL TIME reduction, analysis, & alerts • Send out alerts of transient sources within 60s • ~2 million events per night every night for 10 years • Provide automatic data quality evaluation, alert to problems • Change survey observing strategy on the fly based on conditions, last field visited, etc. Photons to Petabytes 2014

24. LSST:“Data Science” in real time TRANSIENT SCIENCE (Data Stream) • >3 Terabytes per hour (reduced) that must be mined in real time for alerts. • 20 billion objects will be monitored for important variations in real time. • ~2 million events per night every night for 10 years New approaches must be developed for knowledge extraction in real time NON-TRANSIENT SCIENCE • >1010 objects in a 20 PB final database catalog, backed by a 100 PB final image archive New approaches to data mining needed to sift through data to identify samples, or individual objects, of interest Photons to Petabytes 2014

25. Data Management Sites and Centers • French Site • Processing Center • Data Release Production • HQ Site • HQ Facility • Observatory Management • Science Operations • Education and Public Outreach • Archive Site • Archive Center • Alert Production • Data Release Production • Calibration Products Production • EPO Infrastructure • Long-term Storage (copy 2) • Data Access Center • Data Access and User Services Summit Site Summit Facility Telescope and Camera Data Acquisition Crosstalk Correction • Base Site • Base FacilityLong-term storage (copy 1) • Data Access Center • Data Access and User Services Photons to Petabytes 2014

26. LSST Data Management:Baseline Solutions • High-speed connectivity • Mountain to Base: >100 Gbps • Base to Archive: >10 Gbps(hopefully 100Gbps) • Archive to User: variable, UI challenge • Supercomputer processing & storage • Base in La Serena, NCSA, Others? (France?, Brazil?) • 100 PB final image archive • Distributed (Grid) analysis facilities • Petascale DB (~20 PB final catalog) • Based on open source RDBMS Photons to Petabytes 2014

27. LSST: Strategic Partnerships • Distributed Computing Systems • Supercomputer center(s) to provide bulk storage, large scale processing (e.g., NCSA, NLHPC in Chile) • Grid processing, storage, advanced DB • Data Access for member countries/institutions • Connectivity • High-speed Chilean bandwidth (REUNA) • International bandwidth (AmLIGHT, RedCLARA) • Scientific Analysis Challenges: Data Mining & Astro-Informatics or Astro-Statistics • Separating small signals from systematic effects • Automatically finding unique objects: one in billions Photons to Petabytes 2014

28. LSST Outreach Data will be used in classrooms, science museums, and online • Classroom Emphasis on: • Data-enabled research experiences • Citizen Science • College classes • Collaboration through Social Networking Photons to Petabytes 2014

29. Integrated Project Schedule with Key Milestones FUNDING STARTS NOW!

30. The Science of Big Data • Data growing exponentially, in all sciences • Changes the nature of sciencefrom hypothesis-driven to data-driven discovery • Cuts across all sciences • Industry and government face the same challenges • Convergence of physical and life sciences through Big Data (statistics and computing) • A new scientific revolution Data-to-Knowledge Photons to Petabytes 2014

31. Photons to Petabytes 2014

33. Construction NOWFirst light in 2019Operations in 2022