Overview and new developments in Herschel Data Processing

1. Overview and new developments in Herschel Data Processing Stephan OttHerschel Science Data Processing Development ManagerHerschel Science Data Processing CoordinatorHerschel Science Centreon behalfof all contributors of the Herschel missionhttp://herschel.esac.esa.int/HerschelPeople.shtml

2. Overview of Herschel Data Processing Major improvements during the last year What can HSC/NHSC and HIPE do for your Herschel Data? Further plans and milestones What can you do for HIPE? Questions? Overview NGC 1999 observed by Herschel ESA / PACS & SPIRE Consortium, Tom Megeath, NASA/JPL-Caltech/Univ. of Toledo

3. Overview of Herschel Data Processing Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia

4. Overview of Herschel Data Processing • The system combines for the first time data retrieval, pipeline execution and scientific analysis in one single environment • All tools for data reduction and analysis, e.g. also the expert applications for e.g. instrument calibration are part of the Data Processing System. Therefore the community has access to the same system as the instrument experts • The Herschel Data Processing software is coded in Java/Jython to be license free and portable for different operating systems • Formal support is provided for Windows XP, Vista and Windows 7,Linux, Mac OS X 10.5 ("Leopard") and Mac OS X 10.6 ("Snow Leopard") • Herschel Science Centre (ESA), the Instrument Control Centres (HIFI, PACS and SPIRE) and NHSC jointly manage and contribute to the Herschel Data Processing System

5. Overview of Herschel Data Processing • Herschel Data Processing is a major project, with over 200 contributors and currently 60 full-time equivalents working on calibration, coding, documentation, quality control, testing and tutoring • Very distributed development – we work 24/5 (and often 24/7) on Herschel Data Processing System • 300 data processing related software tickets are raised – and resolved – each month • A major HIPE version is released around each three months. This is currently HIPE 5.0 • Minor HIPE versions (currently 5.2) are released around each three weeks. These serve also as the operational version to generate the standard products • All releases can be downloaded via http://herschel.esac.esa.int/HIPE_download.shtml • Latest HIPE developer releases are available via http://herschel.esac.esa.int/CIB_disclaimer.html

6. Major Improvements during the last year

7. Improvements for user tools, HIPE and pipeline system • User friendliness and performance • Pre-installation steps for installer • Notification of updated user releases • HIPE perspectives have been simplified • Autocompletion in the console • Jobs can be shown and cancelled individually via status progress indicator • Option to gather user feedback • Restructuring of documentation • Support of plug-ins • Updates of documentation start-up page, links to data reduction guides from Help menu

8. Improvements for HIFI • Correct pointing information in datasets (aberration corrected) by use of ephemeris information • Source emission is now displayed as Antenna Temperature rather than flux • Noise reduction in Single Point Frequency Switch and Load/Chop modes • Purity of band 5b • Configurable HIFI pipeline • Spectrum Toolbox stabilisation (SpectrumExplorer, SpectrumFitter and all Spectral Tools integrated in HIPE) • Introduction of HIFI Cookbook and continued documentation updates

9. Improvements for SPIRE • Jump detection in signal timelines, allowing better temperature drift correction • New phase correction algorithm improves small-scale spectral shape • Improved spectral cube creation is now done with the photometer naive map-maker • Baseline correction/destriping algorithm. Work on this area is ongoing • Level 1 spectral products now contain all measured extended flux-calibrated spectra rather than the average spectra per detector • Level 2 spectral cubes are now created by averaging flux values within each pixel in the output map • Implementation of icons/browse products for the Herschel Science Archive

10. Improvements for SPIRE FTS products

11. What can HSC/NHSC and HIPE do for your Herschel Data? Rosette cloud, credits ESA/PACS & SPIRE Consortium/HOBYS Key Programme Consortia

12. Herschel Pipeline Processing and Data Quality Control • Pipelines are executed on the ESAC Grid to produce Herschel Products to different reduction levels • Level 0 raw data • Level 1 instrumental and satellite effects removed • Level 2 scientific analysis can be performed • The data are available in the Herschel Science Archive after the processing is finished (usually on the same day of reception of the data from the satellite) • The fast data quality control cycle by the Technical Assistantsand Instrument Calibration Scientists takes a few days • data quality control is a combination of automatic screening and manual inspection • quality control reports are electronically distributed to experts

13. Bulk and on-demand reprocessing • Currently HSC performs a bulk reprocessing of all data taken up to now twice a year using an even HCSS version (HCSS 2, 4, 6, 8, …) • Bulk reprocessing with HCSS 4 is nearly complete • Significant improvements in GRID software and access to the Herschel Science Archive should lead to a much faster reprocessing with HCSS 6 • Automatic quality control is performed using the latest quality control flags • Spot-checks are performed on a few observations • Users can request reprocessing of observations with the latest operational release using on-demand processing

14. Herschel Helpdesk Contact the HSC helpdesk http://herschel.esac.esa.int/esupport/ or the NHSC helpdesk http://www.ipac.caltech.edu/Herschel/helpdesk.shtml for any questions, bug reports and suggestions you have regarding Herschel Data Processing (and beyond)

15. Further Plans and Milestones

16. Next Steps and Schedule March 2011 Release of HIPE 6.0 March 2011 start of bulk reprocessing of all data taken up to now with HCSS 6 14 – 18 March 2011 Data Processing Workshop @ ESAC Spring 2011 Release of eHSA 4.0 ~ May 2011 Release of HIPE 7.0 ~ August 2011 Release of HIPE 8.0 ~ November 2011 Release of HIPE 9.0 ~ March 2012 Release of HIPE 10.0

17. Foreseen improvements for user tools, HIPE and pipeline system • Reduce HIPE freezes and hangs • Improvement of calibration and data reduction pipelines • Improvement of user friendliness • Data manager for Herschel spectra to improve the robustness of spectrum explorer interface • Taskvariable association and useful tooltips • Simplified data input/output. Access to reprocessed data. • Addition of useful example scripts to the build • Harmonization of final spectra and VO. Workflow between Herschel products/HIPE and VO • Combination of data from the different Herschel instruments • Publication quality plots

18. Foreseen improvements for user tools, HIPE and pipeline system • Upgrade to Jython 2.5 • Improved automatic testing to increase confidence in developer builds and speed-up delivery cycle for releases to the community • Improvement of code quality so the system will be maintainable with the reduced manpower we can expect during post-operations • Improvements to quality control framework, especially to support the bulk reprocessing exercises

19. Foreseen improvements for HIFI • Spectral maps (OTF and DBS Raster) • Inclusion of uplink information how the map has been carried out on the sky • Implementation of rotated maps (non-square RA / Dec grid) • Exporting the deconvolved HIFI spectrum to CLASS • Implementation of browse products and postcards for the Herschel Science Archive

20. Foreseen improvements for SPIRE • Baseline correction/destriping algorithm (photometer) • 2nd level deglitching (photometer) • Correction of electrical crosstalk • zero point calibration of scan maps • Bright source mode pipeline (spectrometer) • Documentation how to perform end-to-end processing of calibrated full-range spectra with PACS and SPIRE

21. What can you do for HIPE?

22. Provide your feedback … on HIPE … on documentation … on HIPE crashes

23. Reply to the DP Questionnaire Participate in the Herschel Data Processing Interest Lists and User’s Group. Subscribe at http://herschel.esac.esa.int/Data_Processing.shtml Participate in the HIPE user/developer workshops Collaborate to HIPE development as contributor and tester

24. Your questions to me?