Overview of PACS scan-map pipeline and map-making and problem areas

1. Overview of PACS scan-map pipeline and map-making and problem areas Cardiff, 8 February 2010 Bruno Altieri (ESA/HSC)M.Wetzstein, E. Wieprecht, D.Lutz, P.Popesso (MPE) M.Sauvage, Hervé Aussel (CEA) B.Ali , Cate Liu (NHSC)

2. Talk overview Scan mapping with PACS Issues with scan maps High-pass filter pipeline MADmap Status, prospects, perspectives

3. PACS Photometer : filled-array • Relatively small rectangular field 2x1 footprint, FOV = 3.5’x1.75’ • 2 channels simultaneously imaged (dual-band): • Blue channel 64x32 array, pixel size = 3.2”, 60-85 µmor85-130 µm • Red channel 32x16 array, pixel size = 6.4”, 130-210 µm • PSF FWHM: 5.2”, 7.7” and 12” in the 3 bands. • On-board readout frequency : 40Hz • On-board averaging, downloaded frequency : 10Hz, to stay within allocated 130kb/s rate.

4. Photometer system transmission 70 μm 100 μm 160 μm

5. Scan map orientation • In reference frame “array” in HSpot • α fixed, constraint on β is possible • Selection of homogeneous coverage offered in HSpot. • Oriented in the sky, “sky” in HSpot • β fixed, constraint on α possible • Note: If α=45o then orthogonal coverage has same depth

6. Two pipelines: photProject: High-pass filter + projection MADmap Common pipeline from L0 to L1 with branching to L2. HCSS 2.0: MADmap removed from HSC pipeline Too low quality maps (as cross scan not handled) Memory problems Plan: re-introduce in HCSS 4.0 pipelines current status

7. Turnover loops optimization • “dead time” decreased from 17s to 5s starting OD221 onwards

8. Speed bumps • Scan speed experiences 10s-20s bumps • during bumps attitude unreliable by several arcsec up to tens of arcsec • Tentatively attributed to some bad/rogue Star Tracker pixels • on-going investigation to mitigate effects • Decrease star tracker CCD temperature • suppress: frame filtering • Correct: local gyro propagation

9. Homogeneous coverage • PACS scan map were somewhat inhomogeneous, when in ‘homogeneous coverage’ • inhomogeneous at 45 degrees (~25% level) • but very homogeneous at 135 degrees ! • Due to the combined effect of larger footprint size and tilt angle of 2.5 degrees • Corrected starting OD221 • adjusting the cross-scan step size and detector tilt 45 degrees 135 degrees

10. High-pass filter + Simple projection

11. Why photProject ? • High-pass + phoProject is more sensitive for point-sources. • Running MadMap over very large and deep PACS scan maps (e.g. cosmological surveys) prevented by serious memory problems • MadMap has still problems of residuals for bright point sources • PhotProject takes ~ 10 minutes for 1.5 h AOR with reasonable memory use • However iteration is needed to optimize results

12. photProject scan map pipeline 9 step to apply to PACS photo data cube (frames) • Flag bad pixels • Flag saturated pixels • Convert ADUs to Volts • Cross-talk correction • Pixel timeline deglitching (multi-median resolution) • Flat-field and responsitivity correction: Jy/pixel • Compute ra/dec for virtual aperture (centre) • Scan leg re-centering • Run high-pass filter, to filter 1/f noise • In two or more passes to mask out (bright) sources for high-pass • Project cube onto a grid to get WCS map • Computes the ra/dec for each pixel corners on the fly Note : no timeline deconvolution as for SPIRE Level 0 to 1 Level 1 to 2

13. Artefact of high-pass filteron bright sources

14. Before… after

15. axy geometrical weights • wxy error based weigths • drop size also customizable (75%, 50%, 25%, 12.5% of the input pixel size) • still error propagation not trustable

16. Pipeline/calibration tuning • Bad pixels • Flux calibration / non-linearity • High-pass filter width: • the short the the better the 1/f noise is removed and striping removed • But at too short width PSF becomes distorted • For point-sources : • 15 in the blue ( 10) • 26 in the red (18) • Not suited for large extended emission but maskedHighPass now behaves correctly on extended ojects • Mask sources for high pass • by coordinates/disks • by signal above map noise • Scan leg re-centering • To mitigate PSF degradation/smearing due to SRPE/RPE. • Interferences removal/mitigation

17. Open procesing issues :masked high-pass filtering Masked-high pass filter not in HSC pipeline yet Could be introduced in HCSS 3.0 Investigating automatic way to process in two passes: 1st pass detected source and create mask, used in the second pass. SDP data reduction needed several passes Caveat: mask should be based on final map given by coaddition of all AORs, otherwise the residuals disappear for bright sources but still flux loss might happen for relatively faint sources not detected in the individual map

18. Open procesing issues:deglitching MMT Deglitching deglitches very bright sources Bright source mask to avoid effect  needs input catalog or input map to create mask MMT does not work well in fast-parallel mode Alternative: 2nd order deglitching sigma clipping during projection to get rid of outliers; implemented and tested, but still too high memory consuming (map to cube index) However 2nd order deglitching not fully operational Use cube signal: artifacts on a strong illumination gradient

19. Open procesing issues:pointing Some maps have high absolute astrometry offsets Up to 5-6 arcsec + some unresolved cases up to 17 arcsec for very large maps (solid offset) Still a residual time shift between PACS frames and attitude data D ~ 50ms  shift of about 1 arcsec in odd and even scan legs in opposite directions at medium speed (20”s), PACS frames are leading Shift of 3 arcsec at high speed (parallel mode) Correction calculated through stacking method in odd and even scan legs separately needs external catalog and iteration, thus difficult to insert in automatic pipeline (problem also for MadMap)

20. Open procesing issues:interferences Interferences present in the blue maps: Scan legs affected by interferences flagged and removed during projection Pipeline task to developed for identification and flagging of affected scan legs Single-scan PhotProjmap reconstruction

21. Open procesing issues:processing optimization Still some (slow) tasks to be coded in java photFlagSaturation Pipeline not yet optimized for various iterations in HPF+ photProject branch : the projection matrix is computed too many times: once to get the first object map, once to project it into the object mask for HP, once to do the 2nd level deglitching and once to get the final map. It is always the same matrix...

22. Key point Good collaboration between astronomers/users and s/w developers Defining new functionality requirements Using JIRA system

23. DEEP PACS imaging

24. MADmap for PACS

25. What does MADmap do? Corrected 1/f “striping” MADmap is designed to remove 1/f noise effects

26. What is MADmap? Map-making technique which will produce maximum likelihood sky maps from time ordered instrument data. MADmap uses the data model: d = A(p,t) * S(p) + n(t), Where, d = detector readout, A=pointing matrix, p=sky pixel, S=signal, n=noise and t is time (or index). For all readouts, all detectors, the set of linear equations is too large to invert; thus, conjugate gradient method is used to find the best least-squares fit to the data, given a noise model n(t).

27. Implementation for PACS DP Cal. Files Level 1 Processing* 2nd level deglitching Pixel-to-pixel, module to module and global drift corrections. Pre-processing Frames makeTodArray() Generate TOD runMadMap() MADmap Naive Map Optimal Map * But see caveat about deglitching.

28. Antennae in “blue” with super-resolution

29. Antennae: PACS 3-color composite

30. Use of MADmap in PACS DP MADmap is still an interactive tool. MADmap will produce non-optimal results if: Noise spectrum in data is different from the noise spectrum in cal. files. Correlated signal is not mitigated. Signal drifts (other than 1/f noise) are not mitigated. Preprocessing is critical Calibration blocks interleaved

31. MADmap pre-processing: step 1 Correct for pixel-to-pixel signal offsets. Two options: Use a calibration offset image Subtract MEDIAN of the entire pixel history Both work equally well. Raw Corrected

32. MADmap pre-processing: step 2 Correct for module-to-module correlated signal drifts. A systematic change in the signal level between individual 16x16 bolo modules. Affects the blue channel more strongly than red. Some data appear to not need it After mitigation. Module signal drift

33. Mitigation The drift is in the signal and appears linear; thus simple linear fits produce acceptable results. HOWEVER; not all data appear to require it. MADmap pre-processing: Module drifts: step 3 MEDIAN (module) Time Different colors are different modules

34. MADmap pre-processing (cont.) Global signal drifts. The MEDIAN signal level of the PACS bolometer array decreases systematically in a correlated fashion from start to the end of observation. No thermistors to correct for it as SPIRE Exploring the bad/non-responsive pixels Mitigated by fitting and subtracting baselines. 3 different options are available for baseline fitting. Correlated monotonic drift in the signal over the duration of the observation.

35. MADmap pre-processing: Global drifts The global drift is the dominant trend in the signal time-line. The drift is correlated amongst pixels. The time-line for one pixel mimics the trend of the MEDIAN value of the entire array.

36. MADmap pre-processing: Mitigating global drifts Fit linear or polynomial models to baseline. (1) Baseline determined from minimum value in 1000 readout bins for the entire array. Useful when source signal dominates time-lines (2) Baseline determined from MEDIAN of the entire array for all readouts Useful for very weak sources Each pixel corrected separately. Baseline fit is subtracted from each pixel’s readout. TIME

37. MADmap pre-processing: Mitigating global drifts (3) Subtract MEDIAN values: Useful only when source emission << sky emission Fails for Galactic plane and other bright emission region.

38. long term drift correction: model 1

39. Interactivity Decisions that have to be made only by examining the data: Whether or not to use module-to-module correction? Which option to use for global signal drifts? Column/row drop outs, signal jumps (electronic glitches), etc., will lead to bad baseline fits. Examine fits for sanity checks. Interactive fitting tool ?

40. The other 1/2 of MADmap processing Naive maps: Collapse the time line by averaging signal on sky pixels. Equivalent to ‘projection’ Used as the “first guess” for ... Optimal mapping: Solve for the best sky signal via least-squares fit to the data model. Requires noise model via calibration files. Quality of final maps depends on preprocessing and relevancy of noise models.

41. Noise calibration files We still have much to learn about these. Three generations of cal files available: “old”(v1): pre-launch noise files based on simulations. Correlation length of 100 readouts Still useful for the red filter And often give best result on the blue “current”(v2): first post launch “dark” field measurements. Produce very little optimization at the moment. “3rd Gen”: Second post-launch attempt. To be available in DP shortly Produce much improved results over “current” files.

42. Summary & Known Caveats At the moment preprocessing requires interactivity. Fits must be examined for sanity checks. Point source artifacts. Not entirely understood May be mitigated by larger noise filter lengths, or better projection to TODs MMT deglitching changes the noise spectra and confuses MADmap We don’t really understand why Some residual drifts are still present. Still a lot left to be done: Revisit photometric quality of point sources Investigate “background” from MADmap ...

43. Scan+cross-scan mandatory MADmap to be re-introduced in HSC pipeline if an only if scan+xscans can be handled.

44. MADmap: Herschel vs IRAS Herschel IRAS 160μm 100μm 60μm 70μm Noise filter problem in blue ?

45. M33 HERM33ES OT KP (C. Cramer) 160um

46. MADmap high-res

47. Documentation • API guide (javadoc) • User’s Manual • http://www.herschel.be/twiki/pub/Pacs/PacsMADmap/UM_MADmapDOC.txt • Missing: • Reduction Recipes • Currently provided as scripts

48. Improvements / Areas of Investigation • Stability of INVNTT cal files • Confused by signal drifts • Need to process a “large” sample of observations from start of operations to now. • INVNTT need to be re-computed to 1000-2000 readouts (many minutes) time –scales. • What is the origin of the baseline signal drifts? • Investigate possible temperature drift correction • Evaporator temperature • Bad pixels, not seeing the sky • Are there better baseline removal techniques?

49. Improvements / Areas of Investigation • Mitigation of point source artifacts. • Some good leads have been suggested by the MADmap coders. • Impossible to process 18 hour AORs in the current setup. • Requires use of MADmap2 (C-code).

50. Aquilla