test

1. test Next Generation Sky Surveys: Astronomical Opportunities and Computational Challenges Bob MannWide-Field Astronomy UnitSchool of Physics & Astronomy University of Edinburgh

2. Outline • Survey Astronomy 101 • Next Generation Sky Surveys • Astronomical Opportunities • Computational Challenges • eSI Theme • Summary and Conclusions

3. Observational astronomy

4. Old Style Many small programmes Target specific objects Manual data reduction Data ends up in astronomer’s desk drawer Cold nights in the dome New Style Few large surveys Map large areas of sky Automated pipelines Data ends up inqueryable database Days at the computer Observational astronomy

5. What is driving these changes? • Policy: “common user instruments” • Software & archive part of instrument project • Economics: • More science per night of telescope time • Technology: • Detectors capable of higher throughput • IT can handle the resultant higher data rates

6. How big is a sky survey dataset? 1. How big is the sky? dΩ=sinθ dθ dφ ∫dΩ= 4π steradians = 4π (180/π)2 square degrees = 41,253 square degrees c.f. area of full moon ~ 0.2 square degrees

7. How big is a sky survey dataset? 2. How detailed a map? • Resolution of ground-basedimages limited by “seeing” • “Point-source” disk ~ 0.5 arcsec (1 arcsec= 1/3600th of a degree) • Sample images adequately few 100 million pixels per square degree (i.e. cover full moon with few 10s of million pixels)

8. How big is a sky survey dataset? 3. How much storage? • 2-4 Bytes per pixel adequate for dynamic range • Full sky image map: few x 10 TB • Catalogue ~10% of image size • Full sky catalogue: few TB

9. Comparing survey systems • Figure-of-merit: étendue • Quantifies speed to map a given area of sky to a given depth under fixed observing conditions • Conventional optics: A Ω = A x Ω Field of View Area of telescope primary mirror

10. Three generations of sky surveys • The Photographic Era: 1950-2000 Schmidt Telescope Digitisation SuperCOSMOS: 1 plate = 2GB image, 105-106 objects Ω: huge A: modest 2,500 SuperCOSMOS requests per day Hubble GuideStar Catalog

11. Three generations of sky surveys 2. First Born-Digital Era: 1995-2015 1997-2001: 2MASS (near-IR) 2000-2014: SDSS (optical) 2005-2012: UKIDSS (near-IR) 2009-2015: VISTA (near-IR) Smaller AΩ than Schmidts, but digital detectors much more sensitive that photographic emulsions

12. Three generations of sky surveys 3. Synoptic surveys: 2009-2030 • Map observable sky every few nights: huge AΩ • Pan-STARRS: PS1-2009; PS2-2012 PS4-2015?; PS16-?? • LSST: 2017-2027 LSST Mass production of detectors: can afford to cover large Ω Pan-STARRS: 1.4 Gigapixel camera PS4 LSST: 3.2 Gigapixel camera PS1 SDSS VISTA World’s largest camera in civilian use

13. Three generations of sky surveys:Data Volumes • Schmidt surveys: • ~60 years of observing time • ~10 years of digitisation by SuperCOSMOS • ~20TB of image data • VISTA: • ~20TB of image data per year • LSST: • ~20TB of image data per night for a decade How come? - “Full sky image map: few x 10 TB”

14. Astronomical discovery space Area Temporal Resolution Polarization Wavelength Angular Resolution Depth Different science goals require coverageof different regions of this space Surveys covering a larger region of thisspace can address more science goals

15. Area Area Temporal Resolution LSST Wavelength Euclid Depth Angular Resolution Area Temporal Resolution Gaia Wavelength Angular Resolution Examples • LSST: • Five optical bands • Large area • Deep • ~1000 visits per field • Gaia: • Wide wavelength coverage • Large area • Good positional accuracy • ~100 visits per field • Euclid: • Large area • Good image quality

16. Summary of Survey Astronomy 101 • Systematic survey astronomy > 50 years old • UK world-leaders throughout this history • Progress through advances in detector technology • Photographic Digital Cheap(er) Digital • Multi-dimensional discovery space • Specific science goals target specific regions of it • High-grasp telescopes cover greater volume: more science • Data volumes increasing dramatically • Importance of computation increasing as a result

17. Outline • Survey Astronomy 101 • Next Generation Sky Surveys • Astronomical Opportunities • Computational Challenges • eSI Theme • Summary and Conclusions

18. Next Generation Sky Surveys • Ground-based • Pan-STARRS: PS1, PS2, PS4, … • Dark Energy Survey • LSST • Space-based • Gaia • Euclid • All large international projects • UK share in each would be 10s of £M • Can we afford a significant role in all of them?

19. Outline • Survey Astronomy 101 • Next Generation Sky Surveys • Astronomical Opportunities • Computational Challenges • eSI Theme • Summary and Conclusions Illustrate with LSST

20. Astronomical Opportunities • Survey science is statistical in nature • Describing properties of populations • e.g. clustering of galaxies stellar populations within galaxies • Detecting outliers from those populations • e.g. very distant quasars very low mass stars Needlargesamples Rare Need to sample large volume

21. Science with LSST • Four themes • Probing dark energy & dark matter • Taking an inventory of the solar system • Exploring the transient optical sky • Mapping the Milky Way • Quantity scientific goals from themes • Parameterize survey system • Mirror size, pixel scale, cadence of observations • Optimise system parameters • Ivezic et al: http://arxiv.org/abs/0805.2366

22. Opportunities ~60PB of image data ~6PB of catalogue Catalogue will contain 10 billion stars 10 billion galaxies 1 million supernovae 5 million asteroids New phenomenae! Challenges How to ship and store all the data? How to keep up with data processing? How to find transients in real-time? How to provide data to user community? How to recognise new classes of variable? LSST: opportunities & challenges

23. Example challenges:1. Data Management • Users will want to analyse subsets of LSST data that are too large to download • Must run data analysis code at the data centre • Relational model doesn’t support all sorts of astronomical analysis well • “SciDB”: generalisation of relational model based on multidimensional arrays • Better coupling to analysis code

24. Example challenges:2. Data Analysis • Many classes of transient require rapid follow-up observations for identification • Requirement: issue alerts for transient discovery within 1 minute of observation being made • High-performance data reduction system - both hardware and software: ~2TB/hour data rate • Real-time pattern-matching algorithms, yielding few false positives

25. It’s clear that astronomers need interaction with computer scientists, but is the converse true?

26. Jim Gray’s answer

27. Outline • Survey Astronomy 101 • Next Generation Sky Surveys • Astronomical Opportunities • Computational Challenges • eSI Theme • Summary and Conclusions

28. Three strands • Scientific Prioritisation • We can’t afford significant roles in all surveys:which should we go for? • Data Management • Can we retain the RDBMS-based approach we’re used to? – or do we need “Sci-DB”? • Data Analysis • Can we produce scalable algorithms for the kinds of analysis we want to run?

29. Goals of the theme • Prepare a Road Map for future survey astronomy in the UK for STFC • Identify those computational topics where further R&D is required • Engage computer science community in addressing those problems

30. Summary & Conclusions • Next generation of sky surveys are different in kind • Enabling new kinds of science – time domain • Requiring new computational techniques • To prepare for them the astronomy community must • Agree on its priorities amongst them • Assess the feasibility of the desired options • Identify the problems needing additional R&D • Engage the computer science community in solving them • This Theme should make progress on all these • Many thanks to eSI for giving us the opportunity to do so!