1 / 38

The Dark Energy Survey Data Management System

This project overview details the processes involved in managing and processing data from the Dark Energy Survey (DES), involving cluster counts, weak lensing, large-scale structure, and supernovae analysis. The system handles data from multi-band surveys using advanced techniques, leading to cosmological insights. Key team members and the data processing workflow from exposures to cataloging are highlighted, showcasing the meticulous steps involved in studying dark energy through DES.

kvogel
Download Presentation

The Dark Energy Survey Data Management System

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Dark Energy Survey Data Management System Ignacio Sevilla Noarbe CIEMAT (Madrid) on behalf of the DES Collaboration From photons to catalogs

  2. DES in a nutshell • Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy: • I. Cluster Counts • II. Weak Lensing • III. Large-scale Structure • IV. Supernovae • Two multiband (photometric) surveys: • 5000 deg2grizY to 24th mag ABgriz • 10-30 deg2 repeat (SNe) • Build new 3 deg2 FOV multi-CCD camera, Data management system, improve Blanco facilities Blanco 4-meter at CTIO Credit:NOAO APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  3. Data management of O(PB) data will be carried out by NCSA Credit: Kotwani et al. (2010) APS-DPF 2011 DESDM I.Sevilla

  4. Data management of O(PB) data will be carried out by NCSA Transfer Process Archive Distribute APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  5. Data management of O(PB) data will be carried out by NCSA Transfer CTIO  NCSA 300 GB/night in 18 h Process Archive Distribute APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  6. Data management of O(PB) data will be carried out by NCSA Transfer Process Orchestration: NCSA  HPC nodes Archive Distribute APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  7. Data management of O(PB) data will be carried out by NCSA Transfer Process Archive Results  Archive nodes Oracle DB Distribute APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  8. Data management of O(PB) data will be carried out by NCSA Transfer Process Archive Distribute Through web portals APS-DPF 2011 DESDM I.Sevilla 11/08/2011

  9. NCSA Team Bob Armstrong Bill Baker (Cristina Beldica) Ankit Chandra Greg Daues Shantanu Desai Michelle Gower Robert Gruendl Joel Houston Wayne Hoyenga Chit Khin Kailash Kotwani Terry McLaren Don Petravick (Project Manager) Yuxuan Yang Institut d’Astrophysique Emmanuel Bertin Fermilab Team Huan Lin Svetlana Lebedeva Nelly Stanfield Douglas Tucker Munich Team Gurvan Bazin Art Carlson Joe Mohr (Project Scientist) SWG Development links Mike Jarvis + WL SWG (WL pipeline) John Marriner + SNe SWG (Diff Imaging) Molly Swanson + LSS SWG (Survey Masks) I.S. + LSS SWG (cosmic rays, satellites) SWG Testing links Clusters- Sarah Hansen, Wayne Barkhouse LSS- I.S. DESDM is led by NCSA with multi-national links 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 9/29

  10. CCDs are exposed for ~100 seconds and raw data for all 62 stored in FITS file Exposure consists of 62+ CCD images – 570 Mpix - 3deg2 Survey is ~150,000 100-sec exposures over 525 nights Auxiliary CCDs record images for autoguiding and calibration X 300 + calib. = RAW DATA we send this to NCSA 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 10/29

  11. Data is processed nightly: instrumental effects are removed from images Credit:NOAO Correct for cross-talk among CCDs. Correct for bias levels, non-uniformities, other optical and electrical effects. DETRENDED DATA 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 11/29

  12. Data is processed nightly: the absoluteposition is determined We need reference star catalogs Full focal plane is fit to single solution Correctopticaldistortion at focal plane Credit: E.Bertin We use Sextractor, SCAMP software by E.Bertin 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 12/29

  13. Data is processed nightly: the absolutephotometry is determined Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst): Make big least squares solution for a,b,k; apply results to science images. input output REDUCED DATA 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 13/29

  14. These steps conform the nightly processing • At this point we have, for every night, approx. 300 exposures corrected by: • Instrumental effects • Absolute position • Absolute photometry • This is the nightly processing. • We store these in the archive (+ auxiliary images, info). 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 14/29

  15. Images from different pointings of the same place are co-added to go deeper Go deeper; calibrate better BUT Point spread function is inhomogeneous: each exposure has different quality single exposure single exposure single exposure single exposure single exposure single exposure single exposure COADDED DATA 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 15/29

  16. Images from different pointings of the same place are co-added to go deeper Find PSF in each image Homogeneize per coadd tile 0.77 ’’ 0.94 ’’ 1.32 ’’ 0.94 ’’ PSF HOMOGENEIZED COADDED DATA 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 16/29

  17. Coaddition takes place between observing seasons • It takes 30x more time for coaddition with respect to nightly: • PSF has to be extracted • PSF has to be homogeneized • Actual addition of image and recomputation of errors • Additionally: • Global photometric calibration among all images of the season • We store these in the archive (+ auxiliary images, info). 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 17/29

  18. Cataloging is performed over coadded images This step is performed with the SExtractor package Position, shape and photometry is calculated. E.Bertin CATALOGS 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 18/29

  19. Data will be served to users through catalogs Hundreds of millions of objects with hundreds of columns each: APS-DPF 2011 DESDM I.Sevilla

  20. Data will be served to users through catalogs Hundreds of millions of objects with hundreds of columns each: APS-DPF 2011 DESDM I.Sevilla

  21. Data will be served to users through catalogs Hundreds of millions of objects with hundreds of columns each: APS-DPF 2011 DESDM I.Sevilla

  22. Additional measurements on image through specialized pipelines Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Photoz pipeline (neural network) mag_band_g = 20.7 mag_band_r = 19.2 mag_band_i = 18.5 … + errors, other estimates DESDM (this only one kind of photo-z! More estimations foreseen) 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 22/29

  23. Additional measurements on image through specialized pipelines Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Eliminate instrumental signature DESDM Obtain true shape (intrinsic galaxy shape+ shear) WL pipeline local PSF 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 23/29

  24. Additional measurements on image through specialized pipelines Photometric redshift pipeline: take fluxes in five bands -> estimate redshift Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear. Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena. Credit: Pan-STARRS 11/08/2011 APS-DPF 2011 DESDM I.Sevilla 24/29

  25. Pipeline is tested through the data challenge process as in a particle physics project Produce cosmological simulations Process through atmosphere, detectors, include ‘nasty’ stuff APS-DPF 2011 DESDM I.Sevilla

  26. Science Working Groups study the quality of data through these Data Challenges (~200 sq.deg.) Galaxies Stars APS-DPF 2011 DESDM I.Sevilla

  27. DES Data Management has been put to test using real-world data • Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic2 camera). • Large scale management with SDSS data. BCS Images of First SPT Clusters Credit: Blanco Cosmology Survey APS-DPF 2011 DESDM I.Sevilla

  28. Summary and outlook The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products. Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of 2012. Raw and reduced images to be released yearly, catalogs at midpoint and end of survey. Community pipeline getting ready for usage of the DES camera starting May 2012. APS-DPF 2011 DESDM I.Sevilla

  29. Backup slides APS-DPF 2011 DESDM I.Sevilla

  30. A science portal has been developed to serve as a centralized node for all science codes • Science portal hosted by Brazilian collaborators to offer various analysis pipelines (to run on catalogs, images) and value-added catalogs.

  31. Project Organization • Development Funding ~$6M • $4 million from NSF • $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and Munich • $300K from DES collaboration for Community Pipeline APS-DPF 2011 DESDM I.Sevilla

  32. Science Working Groups study the quality of their analysis pipelines through Blind Cosmology Challenges (~5000 sq.deg.). Carnero et al. 2010 (new simulations with more DES-like systematics coming up by Stanford team)

  33. PreCam uses 2 scientific grade CCDs as a testbed for DECam and calibration purposes

  34. Mini-survey for two weeks for camera and data management validation: first science! Perform full-depth observations in 100 sq.deg. Area is off main survey Overlap existing datasets when possible Run acceptance tests on data

  35. Weak lensing pipeline

More Related