1 / 43

Data Processing Day

Data Processing Day. 9:00 Data Flow System Deliverables and Integration 10:00 Coffee Break 10:15 The ESO Common Pipeline Library 11:00 The ESO-Reflex Environment 12:00 Lunch break 13:00 CPL Tutorial 15:00 Coffee Break / End of Workshop. SDD / Pipeline Systems Department. P.Ballester

chandler
Download Presentation

Data Processing Day

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Processing Day 9:00 Data Flow System Deliverables and Integration 10:00 Coffee Break 10:15 The ESO Common Pipeline Library 11:00 The ESO-Reflex Environment 12:00 Lunch break 13:00 CPL Tutorial 15:00 Coffee Break / End of Workshop

  2. SDD / Pipeline Systems Department P.Ballester K.Banse S.Castro L. de Bilbao A. Gabasch E. Garcia C. Izzo Y. Jung J. Larsen H. Lorch L. Lundin A. Modigliani R. Palsa D. Petry K. Shabun J. Vinther

  3. Data Flow System Deliverables and Integration

  4. Public Release User Community DFS Deliverables (1): Data Reduction Library DICB/Archive Data Flow Operations INS Commissioning Team PSD Paranal Science Operations

  5. DFS Deliverables (2): ETC/ Observation Preparation Tools

  6. Talk Outline • Data Reduction Library • Calibration cascade • Modular design • Data Products • Science-grade data products • Simulations and validation • Project Organisation • Project phases and timeline • DFS Integration • Acceptance tests • DFS infrastructure

  7. Data Reduction LibraryCalibration CascadeModular Design

  8. Astronomical Calibrators (Position, Spectral flux, Diameter, etc..) Instrumental Calibrators (Internal sources) Data Reduction Correct for detector and instrument effects Correct for atmospheric effects Separate science data from noise and background

  9. Calibration Cascade Photometry Standard Star Instrument and Detector Sensitivity Detector Bias

  10. Shift & Add: HAWK-I Modular Design • Independent modular recipes • Integrated pipeline recipe

  11. Shift & Add: HAWK-I Reflex Workflow

  12. Shift & Add: Demonstration • Raw image (sky = 10,000 star = 1) • Dome flat • Estimating the sky from N jittered exposures • Subtracting the sky and correcting for flat-field • Co-adding the sky corrected images • Astrometry + Photometry !!

  13. Science-Grade Data ProductsAccuracyRecipes DesignRobustness, Fault toleranceValidation

  14. Science-Grade Data Products • Calibrated in physical units with error estimates • No residual systematic error • S/N close to the achievable optimum (e.g. ETC prediction) • OB combination • E.g. fully calibrated and mosaiced images with error bars • Standard data formats • SGDPs are independent of the science goals

  15. Understanding the Instrument Signature (1)UVES Adaptive Optimal Extraction • UVES archive data reprocessing: • Robust to CCD defects • S/N adaptive optimal extraction • Analytic profile for low S/N • Non-parametric profile for high S/N • Ripple scale QC parameters

  16. Understanding the Instrument Signature (2)MIDI Visibility Spectra

  17. Inverse Solution Objective function No assumptions on scientific program Optimization strategy Computational complexity Robustness, Fault tolerance Input Data Signal-to-noise Missing data Contamination Stability (Atm., Ins.) Multiplex/Volume Reduced Data Random error estimates Systematic errors Designing Pipeline Recipes Image Model Optical/Detector model Assumptions on stability/reproducibility Accuracy Throughput Robustness Pipeline Recipe Default parameters Fixed calibration data

  18. First light in the lab (VIS arm) Ar lamp - 1” slit - from P. Spano et al. in Merate, July 2007

  19. Using Simulations Ar lamp - 1” slit - simulated by P. Bristow (ESO)

  20. State-of-the-art DRS pipeline Simulated image, multi-pinhole VIS: curved orders, line tilt varying along order

  21. Pattern Matching Physical Model Robustness, Fault Tolerance

  22. Project OrganisationSchedulingValidation and Acceptance

  23. 4 Phases

  24. 1618 Document and Technical Documentation • 1618 Document (currently v.2.0) • CPL Documentation • DICB / VLTI DICD • Gasgano • Reflex

  25. DFS Deliverables: 1618 Template Schedule

  26. DFS Deliverables: 1618 Template Schedule (cont’d)

  27. From Phase A to FDR • Phase A and PDR Preparation • Reuse existing documents as templates • Identify extra needs: observation preparation, visualisation, link to the data analysis? • Learning CPL before the FDR • First recipes in CPL, coding standards, memory management, … • Review existing pipelines, develop prototypes • Coding starts officially only after the FDR • FDR Preparation • Prototype data reduction algorithms (test and simulated data) • New document: Validation and Test Plan • Observatory Pipeline is template based • Further processing requirements ? • Data combination, interactive processing

  28. From FDR to Science Verification • FDR to PAE • Plan more import validation effort for the first recipes, • In general, plan enough time for testing and finalising • Regular intermediate software releases (3 to 6 months), including test reports • Keep improving simulated or adapted data, in sync with instrument schedule • PAE • Plan for the complete set of recipes to be ready at PAE • Validated data reduction algorithms (using laboratory and simulated data) • Have ready a few alternative calibration/reduction methods • Commissioning to Science Verification • Test and validate on instrument and sky data • Identify and solve the unexpected problems

  29. DICB Approval • DPR Keyword Values • Ideally, classification rules are only based on DPR keywords • Valid values are listed in the DICB 4.0 documents • Submit proposal to DICB by FDR before using new values • DICB Validation • No redefinition of existing keywords (database at www.eso.org/dicb) • Keyword names should not be too long • Avoid underscores and special characters • Multi-HDU files • Always the same structure of data for a given data type • Data extensions come first, auxiliary and optional tables afterward

  30. Acceptance Tests • Usage of CPL recipe template • Following CPL coding standard • Usage of external libraries • Namespace protection • Execution Tests • Completeness of the set of recipes and DRL functions • Availability/representativity of test data • Proper execution of recipes • Generation of products • Memory leaks • Unit tests • Documentation • Detailed Validation • Correctness of results • Validation of input • FITS compliance • User-friendly documentation • Data reduction cascade • Unit tests • Performance and Portability • Execution speed • Standard platforms: Scientific Linux

  31. DFS IntegrationObservatory Pipeline Quality Control PipelineDesktop science-grade data reductionMaintenance

  32. Data reduction environments • Observatory Pipeline • On-the-fly data processing (event driven) • Template-based processing • Static calibration database (only certified products are used) • Quality Control Pipeline • Batch processing of complete data sets (all science and calibration data produced by one ESO instrument in one night) • Best available calibrations are used => data must be organized according to the Calibration Cascade • Desktop science-grade data reduction • Modular and additional recipes are avilable • Several front-ends for scripting (esorex), browsing (Gasgano), interactive (Reflex)

  33. Observatory Pipeline On-Line Archive Sytem Archive Instrument Raw Data Shipping Raw data PIPELINE OFFLINE Processed data Further Analysis

  34. Observatory Pipeline: Workstation Reduction Block • List of the raw frames • List of the calibration data • Name of the products • DRS recipe to apply Instrument Package Configuration files Calibration database Data Reduction System PRODUCTS Common Pipeline Library Data arriving from the instrument Pipeline workstation Data Organizer Quality control Reduction Block Scheduler Archive

  35. OCA Rules if DPR.CATG=="CALIB" and DPR.TYPE=="LAMP,WAVE" then { DO.CLASS = "ARC_SPECTRUM"; RAW.TYPE = "WAVE"; } • CLASSIFICATION select execute(GI_WAVE_CALIBRATION) from inputFiles where RAW.TYPE=="WAVE“ group by TPL.START • ORGANIZATION action GI_WAVE_CALIBRATION { select file as MASTER_BIAS from calibFiles where PRO.CATG=="MASTER_BIAS" and inputFile.DET.WIN1.BINX==DET.WIN1.BINX; select file as GRATING_DATA from calibFiles where PRO.CATG=="GRATING_DATA" and inputFile.INS.GRAT.NAME==INS.GRAT.NAME; recipe giwavecalibration { } product mflat { PRO.CATG="MASTER_FLAT"; } } • ASSOCIATION

  36. Quality Control Log Files 12:21:51>-START GROUP / Start [AMBER] 12:21:51> ARCFILE = 'AMBER.2007-03-21T12:16:08.099.fits' [AMBER] 12:21:51> TELESCOP = 'NOT_SPECIFIED' / Telescope [AMBER] 12:21:51> INSTRUME = 'AMBER' / Instrument name [AMBER] 12:21:51> OBSERVER = 'UNKNOWN' / Observer name [AMBER] 12:21:51> PIPEFILE = 'p2vm.fits' / Filename of data product [AMBER] 12:21:51> INS GRAT1 NAME = 'GHR' / Grating common name. [AMBER] 12:21:51> INS GRAT1 RESOL = 527.778; / Encoder resolution [Enc/deg]. [AMBER] 12:21:51> INS GRAT1 WLEN = 2364.972; / Grating central wavelength [nm]. [AMBER] 12:21:51> INS GRAT1 ZORDER = 40319; / Grating zero order position [Enc]. [AMBER] 12:21:51> INS GRIS1 NAME = 'NAR_SLT' / OPTIi name. [AMBER] 12:21:51> INS GRIS2 NAME = '3T_K' / OPTIi name. [AMBER] 12:21:51> INS MODE = '3Tstd_High_K_1_2.365' / Instrument mode used. [AMBER] 12:21:51> PRO DID = 'ESO-VLT-DIC.PRO-1.15' / Data dictionary for PRO [AMBER] 12:21:51> PRO CATG = 'P2VM_REDUCED' / pipeline product category [AMBER] 12:21:51> PRO TYPE = 'REDUCED' / Product type [AMBER] 12:21:51> PRO REC1 ID = 'amber_p2vm' / Pipeline recipe (unique) identifier [AMBER] 12:21:51> PRO REC1 DRS ID = 'cpl-3.0' / Data Reduction System identifier [AMBER] 12:21:51> PRO REC1 PIPE ID = 'AMBER/2.3.2' / Pipeline (unique) identifier [AMBER] 12:21:51> PRO REC1 RAW1 NAME = 'AMBER.2007-03-21T12:13:38.479.fits' / File name of raw frame [AMBER] 12:21:51> PRO REC1 RAW1 CATG = 'AMBER_3WAVE' / Frame category of raw frame [AMBER] 12:21:51> PRO DATANCOM = 14; / Number of frames combined [AMBER] 12:21:51> PRO REC1 CAL1 NAME = 'FlatFieldMap.fits' / File name of calibration frame [AMBER] 12:21:51> PRO REC1 CAL1 CATG = 'AMBER_FLATFIELD' / Frame category of calibration frame [AMBER] 12:21:51> PRO REC1 PARAM1 NAME = 'dummy' / Name of recipe parameter [AMBER] 12:21:51> DET NTEL = 3; / Number of telescopes [AMBER] 12:21:51> QC P1 OFFSETY = 0.02; / Offset wavelength calibration [AMBER] 12:21:51> QC P2 OFFSETY = 0.05; / Offset wavelength calibration [AMBER] 12:21:51> QC P3 OFFSETY = 0.03; / Offset wavelength calibration [AMBER] 12:21:51>-STOP GROUP / Stop [AMBER]

  37. Pipeline @ Garching: QC • Main Tasks • Create master calibrations • Derive and trend QC parameters • Create science products (Service Mode) • Prepare data packages (Service Mode) • Pipelines & QC parameters: requirements & testing • Customers • Paranal Science Operations • ESO community (PIs, archive users) Home Page: www.eso.org/qc

  38. Information by Instrument Similar pages for every VLT/VLTI instrument

  39. Public Releases • www.eso.org/pipelines • Linked from all ESO instrument pages • Release package, documentation, demonstration data 13

  40. Desktop Data Reduction: Gasgano • VLT interactive data organisation tool • FITS file browsing • Grouping • Classification • Interactive front-end • Interface to CPL plugins • Interface to vizualisation tools • Features • Java language • FITS format

  41. Desktop Data Reduction: Reflex • CPL recipes • External tools • Python scripts • Visualisation • Beta users and internal evaluation

  42. Pipeline Maintenance • DFS Tickets • Issued by PSO, DFO, INS, and the user community • Pipeline priority meetings • Bi-yearly meetings with representation of INS, DFO, PSO, SDD • General issues and for each instrument closed, in process, open tickets • Priority setting for further pipeline development • Instrument Evolution • Commissioning of new modes • Detector upgrades

More Related