High Level DESDM Overview and Status

1. High Level DESDM Overview and Status Robert Gruendl (U Illinois/NCSA), Brian Yanny (FNAL)

2. Year 1 – Year 2 Time Line 2013 2014 2015 Y1N: nightly firstcut and SN processing • With early code changes from SVA1 Y1P1: coadd of limited area using early Y1N data Y1A1: integrate changes from Science Working Groups and based on Y1P1 feedback from LINEA Year 1 observations Year 2 observations OBS Y1N processing Y2N processing FirstCut TEST TEST Release SVA1 Y1P1 Y1A1 Y2P1? TEST TEST TEST SWG reports LINEA LINEA SWG??? Feedback CODE FREEZE DES Collaboration Meeting: UIUC 2014

3. Single Epoch (FINALCUT) Processing Changes • Pre-screening of bias/flat exposures prior to supercal production • New BPM on a per supercal basis (thanks to Drlica-Wagner) • New pupil, illumcor (starflat), fringe calibrations • Add named extension: img, wgt, msk • HDU level compression directives • Many external utilities must include CFITSIO v3.36 (or later). • e.g.; funpack, DS9 (treats mask as short rather than ushort) • Crosstalk (updated GAIN and RDNOISE values) • Imcorrect(remove down-weighting of saturated sources) • Astro-refine (thanks to Vikram/Cawthon/Bernstein) • change cataloging call • New distortion correction solution w/ band-dependence • Bleedmask(fix for utility that feeds bleed-trail regions to Mangle) • New Streak-finder utility (thanks to Rykoff/Drlica-Wagner) • Updated cosmic-ray rejection utility DES Collaboration Meeting: UIUC 2014

4. COADD Processing Changes • Add step to allow astrometric refinement of single-epoch images prior to COADD. • Proof of concept complete, for the small number of test cases attempted I am obtaining astrometric solutions with internal residuals of order15-40 mas • Next step is to orchestrate a new COADD pipeline module • Also requires a cut on inputs to remove blacklisted exposures/CCDs • Photometric Calibration slated to use GCM but expectation is alternate calibrations may be available for comparison • (Nebencal: Bauer, YACAL: Annis) • Addition of extinction correction within catalogs • Attempt to run Mangle within COADD pipeline (not yet integrated) DES Collaboration Meeting: UIUC 2014

5. Independent Verification and Validation • Science portal (DES-Brazil) can now ingest and evaluate coaddtile images and catalogs • Also includes ability to ingest tiles from test runs and early tiles from production runs. • For Y1A1 will attempt to extend this to provide near real-time ingestion (i.e., as production occurs) • Evaluation of tiles, is a good overall check on data quality. • Also provides platform by which collaboration at large can examine tiles and provide feedback/note artifacts • Mirror version at FNAL is emerging DES Collaboration Meeting: UIUC 2014

6. DESDM: Data Quality Assessment Data quality is evaluated for each exposure at the end of FIRSTCUT processing with the goal to determine whether each observations meets basic survey requirements. Current assessment is rendered based on calculation of the effective exposure time: Teff= (0.9 k / FWHM)2 ( Bkgddark / Bkgd) (10-2cloud / 2.5) = FeffBeffCeff Current cutoffs used are Teff > 0.2 (gY-band) Teff > 0.3 (riz-bands) DES Collaboration Meeting: UIUC 2014

7. Year 1 (vs. SVA1): Breakdown Teff SVA1 YEAR1 DES Collaboration Meeting: UIUC 2014

8. Year 1 (vs. SVA1): Breakdown DES Collaboration Meeting: UIUC 2014

9. Y1A1 Production Plan • Will (re)process those Year 1 exposures with FINALCUT that passed FIRSTCUT quality cuts and overlap footprint: • Also included are: • Exposures between SPTW and SPTE areas (to aid GCM) • Survey data south of SPTE (near LMC) • SN exposures with seeing better than some (as yet undecided) data quality cut. • Selected fields from SV (currently only COSMOS observations) • COADD for all tiles in contiguous areas with depth at least depth 2 over 75% of tile in at least 2-of-3 of the detection bands (riz) • Some specialized COADD for SN and COSMOS fields • “normal” coadds and • coadds with depth(s) similar to Y1A1 footprint. DES Collaboration Meeting: UIUC 2014

10. Toward Release(s) Tool developed to predict the detailed coverage/depth at each band for each tile. DES Collaboration Meeting: UIUC 2014

11. Projected Y1A1 Footprint Using Year 1 data that has passed FirstCut, Footprint is color coded by depth on a per tile, per band basis. DES Collaboration Meeting: UIUC 2014

12. Y1P1 and Y1A1 Footprint(s) • Y1P1 area included STRIPE82 and SPTW areas but only for observations prior to Dec 1, 2013 with best astrometric residuals. • Y1A1 footprint: • STRIPE82, SPTW, and SPTE • SPT-E may also include southern extension (near LMC) • Depth 2 (or greater): • ~3000 tiles • ~1500 sq degrees • Depth 3 (or greater): • ~50% of area Y1A1_STRIPE82 Y1A1_SPTE Y1A1_SPTW DES Collaboration Meeting: UIUC 2014

13. Science feedback into DESDM Incorporating feedback from DES Science Working Groups back into DESDM production. Processed data DES Science Requirements DES Collaboration Meeting: UIUC 2014

14. DESDM Schedule (with Integrated Schedule items) • SVA1 released: Sep 2013 • First SWG reports: Nov 2013 (IS#180) • 2nd SWG reports: Jan 2014 (IS#180) • Y1(P1) astrometry bug: Feb 2014 (IS#182) • Y1A1 feature freeze: Mar 2014 (IS#161) • Y1P1 release (w/bug): Apr 2014 (IS#94) • Y1A1 code testing: May 2014 • Y1A1 production: May-Sept 2014 (IS#95) DES Collaboration Meeting: UIUC 2014

15. Resulting high priority change requests for Y1A1: Improve astrometric registration of coadds (which leads to improved PSF model,PSF star mags, detmodel galaxy colors). Improve photometry or at least improve the flagging of areas of bad photometry. Improve the flagging of areas with defects(star trails, ghosts, saturated, cosmics, satellites). Control compression algorithms to prevent artifacts from affecting the downstream results. DES Collaboration Meeting: UIUC 2014

16. Detail level #2: The mask (right) does not capture the diffraction/edge-bleed spikes in the actual image (left): Todo for Y1A1: Make the masks reflect this edge bleed effect. DES Collaboration Meeting: UIUC 2014

17. Issue (Galaxy Clusters Group): Odd appearance of non-stars in grid like pattern (image boundaries). Suggests poor modeling of PSF across coadd tile...possible astrometry problem. DES Collaboration Meeting: UIUC 2014

18. Follow-up analysis by Bob Armstrong made the astrometry problem with Y1P1 clear: Thresh_weight issue for spurious detections. SVA1 (ok) CCDNUM=2 Noise detctions Our spec is for these to be < 0.3'' rms, many clearly were not. DES Collaboration Meeting: UIUC 2014

19. Astrometry fix and refinement prior to COADD (Y1A1 proof of concept) Y1P1 z-band Y1A1 (pre-test) z-band Y1A1 (pre-test) riz-band(s) DES Collaboration Meeting: UIUC 2014

20. 5/19/2014 DES Collaboration Meeting: UIUC 2014 20 Refactored System What is 'refactoring’? Dictionary Definition: Dictionary definition: For DES: Improving a computer program by reorganizing its internal structure without altering its external behavior. What it means for DESDM: Changing the 'wrappers' and 'workflow' system that callour underlying software codes (detrend, sextractor, scamp, swarp, supernovae-difference-imaging, etc) to allow more flexibility for making timely changes and allowing portability to other platforms for development and production. DES Collaboration Meeting: UIUC 2014

21. 5/19/2014 DES Collaboration Meeting: UIUC 2014 21 Refactored System What is 'refactoring'? • Design rules: • survey-wide unique filenames; • http[s]:// accessible file trees; • FITS, text file formats; • header keyword metadata; • limited use of proprietary software; • ability to run on generic linux box (compile w/gcc). • Example: Unique filenames for all raw and processed data products: • Allows data to be processed 'anywhere’, then brought home to central archive without namespace conflicts. • Allows same exposure to be processed several times, several ways, and versions of software and parameters used to process are tracked. Dictionary definition: For DES: DES Collaboration Meeting: UIUC 2014

22. 5/19/2014 DES Collaboration Meeting: UIUC 2014 22 Refactored System What is 'refactoring'? A sample unique filename: D00155214_i_c23_r68p01_craymasked.fits D/S = DECamor Simulated data) 155214 = exposure number i= band c23 = ccdnum r68 = request number (software version id) p01 = processing attempt (allows repeat processing) craymasked= reduced image, identifies process step (i.e: cosmic rays masked in image) If image reprocessed, p01 is incremented, If software or processing params change, r68 is changed. Dictionary definition: For DES: DES Collaboration Meeting: UIUC 2014

23. 5/19/2014 DES Collaboration Meeting: UIUC 2014 23 Refactored System What is 'refactoring'? • Status of DESDM refactoring as of May 5 2014: • Basic python wrappers and workflow in place. • Finalcut/Firstcut/SNESE pipeline refactored, tested • Finalcut tested on fermigrid (new platform) • Some http/curl file transfer bottleneck issues • Coadd,diffim pipelines not fully refactored yet. • Expect rapid progress once Y1A1 processing launched • Need to have refactored diffim, proof-of-concept • coadd in place by Y2 datataking start (Aug 2014). • Summer requests: • SN rrep(s) to assist in verifying port of difference imaging pipeline to refactored system • Coadd Q/A checks (compare vs. Y1A1) Dictionary definition: For DES: DES Collaboration Meeting: UIUC 2014

24. Year 1 Data Quality Goals… Year 1 (Y1A1): • Understand extent of telescope and camera defects not fully corrected (flats, ghosting, cosmic rays, amplifier anomalies). • Rudimentary photometry and astrometry (<4% relative photometry, 300 mas astrometry). • Preliminary coadds (demonstrate that noise goes down as 1/sqrt(N), for N exposures of the same field). • Y1A1 shall have these defects mitigated: dome occulsion, shutter stickiness errors, jumps at A/B amplifier divide, satellite trails DES Collaboration Meeting: UIUC 2014

25. OutYear Goals (DRAFT) Outyear goals (beyond Y1), have not been vetted with DES scientists − therefore may be subject to change Year 2 (Y2A1): • Refinement of the fringe correction (z,Y band only) • Remove 'dark corners' from sky subtracted images • Photometry (<3%), astrometry (< 250 mas absolute) • Mask artifacts which affect photometry/astrometry above this level. • Cumulative coadds from all Y1+Y2 data • Register astrometry coadd tile-by-tile (moved to Year 1) Year 3 (Y3A1): • Analysis of tree-ring, tape bump CCD artifacts • 2% photometry, 200 mas astrometry abs, 40 mas relative astrometry DES Collaboration Meeting: UIUC 2014

26. OutYear Goals (DRAFT) Year 4 (Y4A1): • Implement correction for brighter-fatter relation. • Consider tree-ring,tape bump corrections. • <2% photometry across large areas, improved astrometry. Year 5 (Y5A1): • Final algorithms for correcting/masking of artifacts. • Attempt 1% (stretch goal) photometry. • 100 mas absolute astrometry, 15 mas relative astrometry. • Full depth coadds. • Final weak lensing image quality PSFs (needed earlier says WL). Year 6 (Y6Ax): • Best software and Final Releases. DES Collaboration Meeting: UIUC 2014

27. Recruiting: Uncle Rich Need You! Uncle Rich K. says: “We need you to help DES tend it's science software and we'll grow a fine garden of Dark Energy results” DES Collaboration Meeting: UIUC 2014

28. Recruiting: Uncle Rich Needs You! • Brighter/Fatter • Refactored difference imaging and SN single-epoch pipelines • Dynamic masking of CCD 31 • Refine crosstalk correction and masking • Scale fringe and pupil corrections more optimally • (Re)implement single-epoch photometric calibration • Multi-epoch cosmic-ray rejection DES Collaboration Meeting: UIUC 2014

29. 5/19/2014 DES Collaboration Meeting: UIUC 2014 29 DESDM software working sessions this week! What is 'refactoring'? “DESDM future software features” Wednesday 2pm NCSA 1030 Led by Robert and Brian When will we tackle 'brighter-fatter, tree rings, photo-z, forced photometry, etc’? Your chance to request a feature or complain about a bug. “Getting the most out of DESDM Databases” Thursday 9am NCSA 1122 Led by Todd Tomashek All your SQL and DESDM specific query questions addressed. Information about indicies, joins, metadata, creating your own tables. “DES Science Portal” Wed 3:45pm Value Added Catalog prep NCSA 2000 How to get your VAC in/out of the portal Thur10:45am VAC publication and distribution Dictionary definition: For DES: DES Collaboration Meeting: UIUC 2014

30. Extra Slides DES Collaboration Meeting: UIUC 2014