Archive, Data Products, & Operations for the Murchison Widefield Array

1. Archive, Data Products, & Operationsfor the Murchison Widefield Array Ron Remillard (MIT); MWA Project Meeting, June 6, 2011

2. Outline • Resources for the MWA Archive at MIT • NSF MRI Program • Standard Data Products • Incomplete definitions in early 2011 • Strategy for progress • Information Flow Diagram • Commissioning Phase • Needs for Commissioning Plan • Needs for more sophisticated Monitoring • Operations Phase • Operations Plan and Needs for Update • Plans for remainder of 2011

3. MWA Archive at MIT • NSF MRI Program ($868 K: NSF + MIT contribution) • Large equipment budget • Advanced spending on archive requirements, data products • Need to set aside majority of funds until late 2012 • Archive Tasks (based on EoR, systematic error analyses, student projects) • Capture raw data (in short bursts) • Intermediate data capture (non-calibrated images; less resolution (n, t)) • High timer resolution calibrated data • Determine calibrations in parallel with RTS • Calibrate data offline • Study ionosphere parameters at 2 s • Service MWA EoR Consortium, other EoR users, other outside projects

4. MWA Archive at MIT • Gaps in Plan • Redefine SOW for 128T (retain EoR focus; gain approval for NSF extensions) • Complete Archive Requirements Document (project-wide scope) • Method to obtain data from Curtin • Method to provide data access for Users

5. Standard Data Products • Science Data • Eor Data Products (calibrated raw images) defined in MRI proposal • SHI Needs raw data (pro-Correlator) and UV data from outlier tiles • Transients & GEG groups need Images and other modes (light curves, transient searches (image diffs), wider images (low t res.), others?) • “Meta Data” • RTS {calibrator catalogs, calibrator solutions, ionospheric params., “peeled” sources} • M&C {configuration parameters, hardware status, housekeeping data} • Common RTS/M&C Database

6. Standard Data Products • EOR Data Writing Requirements for 512T • Images: HEALPix files, instrumental images (from back end of RTS): • 25o radius ; 4 polarizations; 768 channels • UV data samples: (from front end of RTS): • visibilities for 100[TBR] Tile pairs •  Foundation for MWA Calibrations and Data Processing • High-Quality Calibrations and Supression of Systematic Errors • Calibrator gain measurements; Ionosphere refraction data; • Faraday rotation; beam fit parameters • EOR Data Products are Basis for 3 Other Science Packages • Imaging mode with variable {time, frequency-binning; spatial selection} for Transients Imaging (TLA mode) and GEG Imaging • SHI: UV Data Samples + IPC Mode to be written Directly from Correlator • Stand-alone processing for Transients discovery (Transient ASM mode) • and other data products (e.g., flux-calibrated images; light curves)

7. MWA Information Processing of EOR Data M&C Housekeeping Configuration Receivers Command Writing mode Digitized Data RTS/RTC CalibrationSubtract Calibrated Staging & Solutions Bri. Sources Images Processing Correlator UV Data (40 Gbs) MWA Data Archive (Perth) Catalogs: Calibration Sources Sources Subtracted Raw Data: UV Data [~100 tiles] Science Data: EOR Images Secondary processing Calibration Data: Beam Parameters Ionosphere Params. M&C MetaData: Configurations Housekeeping Flux Images Light Curves (study systematics) Additional MWA Archives (MIT, Melbourne)

8. MWA Information Processing and Data Products M&C Housekeeping Configuration Receivers Command Writing mode Digitized Data RTS/RTC CalibrationSubtract Calibrated Staging & Solutions Bri. Sources Images Processing Correlator UV Data (40 Gbs) MWA Data Archive (Perth) Catalogs: Calibration Sources Sources Subtracted Raw Data: SHI: IPS Data [512] UV Data [~100] (for SHI & EOR) Science Data: EOR Images TLA Images GEG Images Secondary processing Calibration Data: Beam Parameters Ionosphere Params. M&C MetaData: Configurations Housekeeping Flux Images Light Curves Transient ASM Data Additional MWA Archives (MIT, Melbourne)

9. Archive & Standard Data Products for 128T • Save Correlator output (larger Archive)? • Built-in Capacity for UV Data analyses, Iterative Calibrations, Comparative Tests, Specialized Tests, Secondary Data Products, etc. • Standard Data Products: No Significant Changes  Next efforts: Finalize design and track development of standard data products

10. Commissioning Phase (Checkout to Verification) • Requirements & Verification Document • Colin & RR first draft March 2011 • Needs Adaptations for 128T • Needs Attention and Inputs from Judd, Frank, etc. • Gaps and Risk • Verification Details are complicated; plans incomplete • Insufficient strategy for automation and scripts • Upgrade Plan for Monitoring Functions • Status screens for hardware & housekeeping data for Installation/Checkout • Well-defined metrics and useful error logs • Take next step to rudimentary Interferometry monitoring • Add smart scripts to monitor for Intermittent problems • Examine all data • Who? (Curtin Archive & Superscience help? ; MIT student project?)

11. Operations Phase • Operations Management Plan • Last update: Nov 2010 (Wayne Arcus) • Operations succeeds end of Commissioning phase • Engineering component – Dave Emrich (Commissioning Engineer) • Science Performance Verification – Judd Bowman (Project Scientist) • Initial Operations Manager – Tom Booler (Deputy Project Manager for Australia) • Assumptions: move toward 24/7 operations ; proposals ; review ; schedule ; observe ; trend analyses ; archive verification ; TBD user support • Gaps and Risk • Currently No Operations funds • Details for Proposals, Target Selection, Scheduling • Plans during remainder of 2011 • Revise Plan after work on: schedule, cost, monitoring, automation, data products, commissioning plan