1 / 21

Data Processing

Data Processing. Progress and Planning June 2014. Jeff Kern. Data Processing Group Lead. Summary Status. CASA-4.2.2 brings capabilities desired for cycle 2 data reduction Currently under test issues with plotMS , otherwise looking good Key deliverables: Weight handling (per SPW)

rae
Download Presentation

Data Processing

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Processing Progress and Planning June 2014 • Jeff Kern • Data Processing Group Lead

  2. Summary Status • CASA-4.2.2 brings capabilities desired for cycle 2 data reduction • Currently under test issues with plotMS, otherwise looking good • Key deliverables: • Weight handling (per SPW) • Flag Data Improvement (Tbuff parameter to avoid quacking) • Strict Flagging in Applycal • Single Dish Interface Update • Pipeline • Provisional Acceptance in March • Long punch list, mostly cosmetic. Largely addressed at this point • Doing integration testing against CASA-4.2.2

  3. Planning: Near Future • June 27: Prerelease of CASA-4.2.2 and Pipeline to PLWG for testing • July 1: Release of CASA-4.2.2 (Public) • Both OSX and Linux • ~July 1: Release of ALMA Pipeline to project • Based on testing from PLWG • ~July 15: CASA-4.2.2-Pipeline release available to public These dates are all very tight, and are based on not finding issues in the testing. Small slips can have large impact as we are in vacation season

  4. CASA Planning • September 2014 CASA-4.3 • Weight Spectrum • Imaging Refactor V1 • Parallelization (Calibration) • SetJy Improvements • VO Viewer • March 2015 CASA-4.4 • Imaging Refactor V2 • Support for Pipeline • OTF Cal Application • Pipeline Parallelization • Enhanced Phase Connection

  5. Pipeline Planning • From ObsMode Meeting • Support for Cycle 2 Operations • This could stop all other progress! • Data Flagging Heuristics • Bandwidth Switching Heuristics • First of low SNR Heuristics • Science Target Imaging • 1 Cube per Science SPW per Science Target • In parallel: • Automated trigger and cluster exectuion • This is NOT a magic bullet.

  6. Weblog Example Pipeline Regression Server

  7. PLWG: User Test • Pipeline Working Group will perform a user test to receive feedback on Weblog • Format TBD either special run and request comments or just survey of Cycle 2 Pis • Will need to prioritize feedback from this vs. functional testing

  8. Pipeline Single Dish • Well developed and tested • Not much knowledge outside of NAOJ so far • Hoped to be doing parallel testing at this point • Need lots of data (Project wide issue) • Concerned about connection between manual reduction, script generator, and pipeline ICT CPM-3 June 17-19 Socorro New Mexico

  9. Pipeline Releases • November Update • Bandwidth Switching • Data Flagging • Other Cycle 2 Experience (TBD) • March 2015 • Imaging capabilities V1 • Long Baseline capabilities • Low SNR (?) • Q3 2015 • Continuum Subtraction • Self Cal • Other Calibration Infrastructure & Automation Preliminary

  10. Steps to Pipeline Use in Operations Cycle 2 Phase C testing Test commissioned observing modes for Cycle 2 data PWG full acceptance testing Test full acceptance criteria Pipeline documentation review Send final drafts broad review within Observatory PWG full acceptance recommendation Submit report to SciOpsIPT June/July 2014 Test full acceptance criteria implementation Test commissioned observing modes for Cycle 2 data Submit final drafts to SciOpsIPT and DRM WG for review Submit full acceptance report to Observatory First Pipeline-assisted deliveries DRM WG end-to-end testing Test e-2-e QA2 process, packaging & documentation Pipeline-assisted QA2 in operations Pipeline released to users Test e-2-e QA2 process, packaging & QA2 documentation For commissioned observing modes, data calibrated by Pipeline and imaged manually CASA 4.2.2 tarball including Pipeline

  11. Pipeline Issues • Staffing: • Pipeline has exactly 1 full time developer • Currently stepdown another 0.5 FTE at end of year • Formally 3.0 -> 2.5 but realistically 1.8 -> 1.3 on interferometric • “Borrowing” expertise from CASA but not the same as dedicated staff • Maintenance / Support • Tracking down errors for failed SBs could halt progress on new targets • Need robust QA0 process at observatory • Balance between Cosmetic and Functional requirements • Weblog has grown very large, suspect it may be overspecified at this point

  12. CASA and CASA Based Pipelines Pipeline Weblog: Home Page

  13. CASA and CASA Based Pipelines Pipeline Weblog: Observation Summary

  14. CASA and CASA Based Pipelines Pipeline Weblog: Topics Page

  15. CASA and CASA Based Pipelines Pipeline Weblog: Task Summary

  16. CASA and CASA Based Pipelines Pipeline Weblog: Task Detail

  17. CASA and CASA Based Pipelines Pipeline Weblog: Plot Details

  18. CASA and CASA Based Pipelines Pipeline Weblog: Detail Zoom

  19. CASA and CASA Based Pipelines Pipeline Weblog: Imaging Summary

  20. CASA and CASA Based Pipelines Pipeline Weblog: Imaging Details

More Related