1 / 13

SD Pipeline Heuristics Status Report

SD Pipeline Heuristics Status Report. ALMA Project, NAOJ George KOSUGI. E2E Pipeline Process Flow. Reference to Science Data Ref. to Calibrator in a necessary condition need to search other calibrator for the sufficient condition? Pipeline Parameters. Recipe. Scheduler. Pipeline.

elwyn
Download Presentation

SD Pipeline Heuristics Status Report

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. SD Pipeline Heuristics Status Report ALMA Project, NAOJ George KOSUGI ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  2. E2E Pipeline Process Flow • Reference to Science Data • Ref. to Calibrator in a necessary condition • need to search other calibrator for the sufficient condition? • Pipeline Parameters Recipe Scheduler Pipeline Archive ASDM (with raw data) Current Development Offline Scripts ? Filler Pipeline MS2 Telcal Result Engines ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  3. Current SD Pipeline Flow Chart MS for input data AIPS++Table for Info Storage SDFITS Extract Data numarray Text Table Pointing, StartTime, IntegrationTime, etc. Extract Info MS for output > Single Pointing > Multi Pointing > Raster/OTF MS2 Pointing Pattern Analysis Re-Gridding BaselineFit Integration over Channels FlagSpectrum FITS Map numarray FITS for image ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  4. Environment for SD Heuristics Pipeline • Python Packages • casapy with ASAP • numarray: work area for spectrum data analysis • Scipy: Kmean clustering algorism • Numpy: data for Kmean • PyLab: plotting • PyFits: for collecting info from SDFITS header ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  5. What is necessary for SD UT • MS input • Need sample MS • MSs for casa/ASAP UT in 2007/03 will be adequate • Store/Retrieve info into/from AIPS++Table • MS and FITS output ----------------- 2~3 FTE-w • Calibration script • Investigate TelCal Engine • Investigate scripts used in the casa/ASAP UT ----------------- 2~3 FTE-w ? • Additional plots ----------------- 1~2 FTE-w ? ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  6. Baseline Fit is done for each spectrum one by one • Determine Line Free Channels (if not specified) • Fit and Subtract Baseline Calculated from the Line Free Channels ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  7. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  8. Determine Line Free Channels (if not specified) • Subtract DC (bias) and gradient components • Accumulate several spectra in close vicinity in space to enhance S/N ratio for line detection • Line Detection by application of ASAP.asaplinefinder • Kmean clustering analysis is applied to the detected lines (parameters are line center and line width) • Select real (supposed to be) lines by examining continuities in space, distortion of the vector, density of the detection, and cluster size • Calculate 2-dimensional fit (ra, dec, line center), (ra, dec, line width) to each cluster member • Apply fit to spectra whose position is within the clustering radius ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  9. Fit and Subtract Baseline Calculated from the Line Free Channels • Determine fitting order for polynomial • Search gaps in the time sequence • fitting order is determined for each bunch of data (between 2 time gaps) • fitting order is determined from frequencies of major components in the Fourier space • Calculate Fit • to avoid jaggy due to the shortage of the accuracy in the calculation of higher-order power for polynomial, spectrum is divided into sub-spectra, if necessary. • ASAP.asapfit was used for the fit, however it didn’t converge sometimes. So far python script is used for polynomial fit. ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  10. Spectrum based Flagging (not channel based) • Statistics are measured for each spectrum (emission line channels are excluded): • baseline rms • before baseline subtraction • after baseline subtraction • difference from the running mean • before baseline subtraction • after baseline subtraction • may need other criteria • Tsys, etc... ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  11. Plot statistics ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  12. Gridding (Align to Lattice pattern) • Beam size and Grid spacing are given • Spectra at positions inside beam radius from the grid center are combined into one spectrum • weight: none, gaussian beam, or cone-shaped beam • combine: median (no weight), weighted mean, NmaxMmin rejected weighted mean ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

  13. Line Intensity Map • Accumulate flux in given channels • Subtract flux outside given channels ALMA Pipeline Heuristics F2F Meeting 2006 in Paris

More Related