1 / 21

VIRCAM & CPL: Lessons Learned

VIRCAM & CPL: Lessons Learned. Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit. Introduction to CASU. Small group within the Institute of Astronomy, specialising in survey astronomy. Pipeline reduction of imaging data APM (Schmidt Plates 1 x 40k x 40k)

ezra
Download Presentation

VIRCAM & CPL: Lessons Learned

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. VIRCAM & CPL: Lessons Learned Jim Lewis and Peter Bunclark Cambridge Astronomy Survey Unit Second Generation VLT Instruments

  2. Introduction to CASU • Small group within the Institute of Astronomy, specialising in survey astronomy. • Pipeline reduction of imaging data • APM (Schmidt Plates 1 x 40k x 40k) • INT Wide Field Camera (4 x 2k x 4k) • ESO WFI on 2.2m at La Silla (8 x 2k x 4k) • MOSAIC-1 on KPNO 4m (8 x 2k x 4k) • MOSAIC-2 on Blanco 4m at CTIO (8 x 2k x 4k) • AAO WFI on AAT (8 x 2k x 4k) • CIRSI on INT (4 x 1k x 1k) • INGRID on WHT (1k x 1k) • UFTI on UKIRT (1k x 1k) • WFCAM on UKIRT (4 x 2k x 2k) Second Generation VLT Instruments

  3. Introduction to VISTA and VDFS • VISTA: 4m survey telescope on Paranal • VIRCAM: The IR camera for VISTA • 16 Non-buttable 2k x 2k detectors • VDFS: PPARC funded facility to provide an end-to-end data-flow system for VISTA and WFCAM. • Quality control and calibration pipelines • Paranal and Garching • Science pipeline for full calibration of science data. • Cambridge • Science archive acts as the point of access of the reduced data. Plus some further processing. • WFAU, Edinburgh Second Generation VLT Instruments

  4. Data Flow • Raw telescope data is assessed by the summit pipeline (QC1) • Shipped to Garching (discs) • Shipped to CASU for science reduction and calibration. (discs) • Calibrated data shipped to Edinburgh for archiving (ftp) Second Generation VLT Instruments

  5. IR Data Reduction Worries • IR detectors are currently inherently more unstable than optical CCDs. • Some odd electronic effects • Much poorer cosmetic quality than CCDs • Sky emission > 100x brighter than most objects • And it’s variable both spatially and temporally! • Exposure times are short, so data rates are very high. • 200-500 Gb/night expected for VISTA public surveys • Rice tile compression can save factors of 3-4 in 32 bit integer data Second Generation VLT Instruments

  6. VDFS Pipeline Recipes • Create master calibration frames (dark, twilight flats, confidence maps, etc) • Linearity analysis • Detector noise & dark current properties • Persistence and crosstalk analysis • Illumination correction analysis • Full reduction recipes for standard star and programme fields Second Generation VLT Instruments

  7. Summit & Garching Pipelines • QC1 parameters • e.g. photometric zeropoints, astrometric fit quality • Written using ESO qfits/CPL infrastructure • Both use the same software modules • Amount of processing can be scaled down • Calibration images (flats etc) from a master library. Second Generation VLT Instruments

  8. Cambridge Pipeline • Full reduction • e.g. better sky correction • Results shipped to WFAU • Some of the same CPL based modules reused. Some additional modules required. Second Generation VLT Instruments

  9. Current Status of VISTA Pipelines • Version 0.5 released to ESO in March • Full compliment of recipes (almost!) • PAE Second Generation VLT Instruments

  10. The Decision To Use CPL (1) • We were given the choice. • We already had advanced pipeline software in our own environment for WFCAM and other IR instruments. • First view of CPL and qfits was not favourable. Second Generation VLT Instruments

  11. The Decision To Use CPL (2) • CPL v2.0 was a great improvement • CASU has an ‘ESO workstation’ to generate ‘data’ using the VIRCAM acquisition software • Integration into Paranal and Garching systems greatly simplified by using CPL Second Generation VLT Instruments

  12. Things I wish I had known… Second Generation VLT Instruments

  13. My First Attempt At A Recipe • INPUT: • FITS image of WFCAM data • Binary FITS table of astronomical objects • Binary FITS tables containing the 2MASS catalogue • TASK: • Work out RA, Dec coverage of image using header FITS WCS keys • Extract 2MASS standards • Cross match standards to object catalogue • Fit a WCS • Write resulting WCS to FITS image Second Generation VLT Instruments

  14. World Coordinate Systems • CPL has no WCS facility • But it soon will Second Generation VLT Instruments

  15. Input File Access • CPL doesn’t give write access to input files. • cpl_frame, cpl_image • Data abstraction is not perfect • cpl recipes can’t be written to update a file Second Generation VLT Instruments

  16. CPL vs FITS tables • CPL does not allow for direct manipulation of FITS tables • cpl_table structures can be created from a FITS table. • But one must read the WHOLE table even though you might only be interested in a few rows or columns. • Tables with many rows take too long to read • Break them up • Tables with many character type entries cause the system to run out of memory pointers • Remove unnecessary character columns Second Generation VLT Instruments

  17. User Documentation • Reference manual, users guide, software developers kit • User’s guide and SDK are helpful but limited in scope • Very few examples • Get a copy of a working CPL pipeline Second Generation VLT Instruments

  18. QC • Garching and Summit pipelines generate QC parameters • Provides data for QC trending to be done both on site and in Garching • The ‘ESO System’ is really two systems • QC passed through FITS headers (Garching) • QC passed through PAF files (Chile) • Neither system appears to be documented. • PAF files contents are not documented in CPL users or reference manuals. • CPL has no standard class for passing QC parameters Second Generation VLT Instruments

  19. File Tags • DPR TYPE • OBSTYPE • PRO CATG • Product type • No two recipes can produce the same value • DO CATG • File type for files used as input to recipes • Raw frame DO CATG values must be unique to each recipe. • There are some rules on the values here. Second Generation VLT Instruments

  20. Output File Names • Recipe creates and names output files. • File names must be hard coded! • Several files of the same PRO CATG must be indexed in a predictable way. • esorex can change the file name Second Generation VLT Instruments

  21. Headers • CPL cannot directly manipulate FITS headers • These must be read into a cpl_propertylist structure. Second Generation VLT Instruments

More Related