ALMA Computing IPT Science Software Status Update

1. ALMA Computing IPTScience Software Status Update Gianni Raffi, ESO ALMA Science Advisory Committee 15-16 September 2006 ASAC Meeting- Computing IPT

2. 2008 2009 2006 2007 2010 2011 2012 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 1 2 3 4 IR 6.0 (CPA punch list) IR 5.1 (Interferometer support) IR 4.1 (Single antenna commissioning) IR3.0 IR 4.0 (Single antenna modes) CDR 7 CDR 8 PAR R6.0 R5.1 R4.1 R7.0 R9.0 CPA #1 R4.0 CDR 4 R3.0 CPA #2 RR IR 7.0 (Commissioning feedback) Interferometry at ATF 1st Antenna Integration 3 Antennas Integrated IR9.0 Full Science Support 1.07 Computing Summary Schedule (Data from IPS as _________) Time Now Array Software >> Observatory Software >> Computing IPT Management and Science Software Requirements High Level AnalysisandSoftware Engineering Common Software and Archive Executive, Correlator and Control SW Schedule, Observation Prep, Offline, and Telescope Calibration Software Integration and Test ASAC Meeting- Computing IPT

3. Requirements Management Goal:: • Evaluate development progress with respect to requirements • Provide reporting. => Done via Central database (DOORS) with requirements derived from all sources (science, technical, operational) that allows us to: • Manage requirements (addition, editing, deletion, prioritization, review) • Assess the status of software development (reporting) • Analyze the impact of requirements changes • Coordinate requirement tracking for all computing subsystems. • Complete functional team work and release development with integration tests (by ITS team) • Validate implementation with user tests (by SSR subsystem scientists) following each release. ASAC Meeting- Computing IPT

4. SSR Testing of Science Software Extracted from D.Shepherd’s page on TWiki at: http://almasw.hq.eso.org/almasw/bin/view/Usertests/WebHome#Test_Plans Requirements available on SSR TWiki (unless explicitly mentioned below) All functional subsystems shown for completeness (except ACS framework and ObOps) • EXEC • R3.1 - Executive internal pre-test complete. • R4.0 - user test scheduled to begin October 20. • Scheduler • R3.1 - Optical pointing test completed in March 2006 • R4.0 - Testing planned for early fall 2006. This will test the scheduling simulator, particularly with weather simulation. • TelCal • R3.1 - Features developed during this release tested August 21, 2006. Optical pointing reduction tests completed March 2006. • R4.0 – holography reduction tests scheduled late 2006. • Archive • R3.1 – Basic archive function tested during optical pointing integrated user test in March 2006. • R4.0 – Tests of archive queries (projects, monitor point data) scheduled for integrated user test late 2006 • Requirements: Use Case development for Science Archive continues. Granular requirements for Archive starting to be produced. ALMA Science Archive (ASA) requirements about ready. ASAC Meeting- Computing IPT

5. SSR Testing Status (cont.) • Correlator • R3.0 - Testing planned for April, June, and July slipped to October 2006. • Pipeline • R3.1 - End to end user test (from flagging to imaging) is planned for the pipeline in Fall 2006. • Quick Look: R3.0 - Operations test should be possible. It is not yet planned. • ObsPrep • R3.1 – User test about to conclude. • Control • R3.1 – Optical pointing features associated tested during integrated user test in March 2006. • R4.0 – Holography features scheduled to be tested late 2006 during integrated user test. • Requirements: adopting EVLA control requirements (modified for ALMA) for ALMA control granular requirements. Passed System Requirements Review. • Offline • User Test 4: R2.1 - Single baseline data reduction and analysis - supporting the ATF. Test complete, report written. • User Focus Group: completed March 2006, provided input on python user interface. • User Test 5: Will test single dish reduction & imaging. Scheduled for Dec 2006 – date dependent on ACA requirements. Note: Control and correlator are exercised not only at ATF, but also every day in the Prototype System Integration (PSI) lab. ASAC Meeting- Computing IPT

6. Function Based Team (FBT) development • Works to functionality across subsystems (~2 months) • Based on Technical Use Cases • Interface issues clarified soon • Allows more frequent integration • Some parallelism possible In principle planning of R4.0 FBTs ASAC Meeting- Computing IPT

7. Latest Developments • Concluded R3.1 release and integrated in April • Includes Optical Pointing e2e functionality (tested in March at ATF) • Installed and tested on both prototype antennas • Installed also in the Prototype System Integration Laboratory in Socorro • Included integrated regression tests • FBTs so far: • Optical Pointing (completed in March) • Shared Simulation of Single Field Interferometry (SFI) (April) • Monitor DB optimization (July) On-going FBTs: • ALMA Project Data Model (APDM) re-factoring (Aug) • Validation almost complete. Currently on hold behind higher priority Holography work. • Holography (Aug). Hw for tests in the lab early Sept., at ATF late Sept. • Single Field Interferometry II (to be used at ATF just after first fringes- achieved with PSI scripts) ASAC Meeting- Computing IPT

8. OPT User Test (mar.’06) (1st integrated user test of ALMA software) • evaluated by members of the Computing IPT (CIPT) in March 2006. • primary software limitations were found to be jlog instability and slowness, lack of coherent logging levels across subsystems, archive access (query mechanism not documented at the level of end users), lack of access to archived monitor points, missing CCL offset commands, and uncaught exceptions (e.g. Control, TelCal) that caused software crashes. • Many of the interfaces and system responses have been designed for the developer. • Approximately 170 defects, issues, and enhancements were identified in this report. Roughly 40% (70) of those are defects (14 (8%) priority 1 and 56 (32%) lower priority). ASAC Meeting- Computing IPT

9. Next Developments (*) • R4.1 ( up to June 2007): • Single Dish (SD) Function Based Team (Feb.) (w/o nutator) • TP mode 1: OTF mapping in continuum Total Power • TM mode 2: radio pointing measurements using TP • Test SFI with h/w at ATF (completion of SFI II work) • Readiness Review (external) by subsystem (May) • R5.0 ( up to Dec 2007) • SD: Nutator, Spectral line (prepare for equipment in Chile) • Interferometry: WVR, fast switching, Pointed mosaics • Respond to AIV use of software in Chile • PAR (Dec.07) – Delivery of modes developed thus far to Chile • During all of 2007 support in Chile for antenna: • Acceptance and integration • System characterization using continuum TP (*) Development plan will be adapted to availability of h/w         CIPT formal user test will occur twice next year ASAC Meeting- Computing IPT

10. Archive Software Persistency support for raw data, pipeline results (images and logs), SBs and all other User input, site characteristics, monitor data, calibration data, catalogs, and all other auxiliary information required for ALMA to observe. Quick look results are temporarily stored in the Archive. Items not specific to ALMA Science Archive: • User Repository: Manage database of registered users, support proprietary periods. • User query, retrieval and monitoring capability. (event data R4.0) • Data replication between OSF, Santiago, ARCs (both via network and physical media). • Use of NGAS supported data replication. Requires ALMA testing. • Request Handler: Manage packaging, request tracing and delivery of data products to the scientific user community (implementation planned for Q4 2007) • High-availability system with failover (test implementation Q3 2007) • Provision for safe backup of all data; when more than one Archive is operational this may be accomplished via data mirroring between Archives. • Support 6 MB/s average and 60 MB/s peak (sustainable for many hours at a time) data rates. ASAC Meeting- Computing IPT

11. Archive Replication between SOC and ARCs Full description on Archive Twiki at: http://almasw.hq.eso.org/almasw/bin/view/Archive/DatabaseReplication • Bulk data replication taken care of by NGAS. • 2. Transaction based replication for Oracle DB OSF-SCO • 3. Standard DB incremental backups sent via NGAS. ASAC Meeting- Computing IPT

12. ALMA Science Archive (ASA) • Scientific view on ALMA observations through ALMA data products (cubes). • Preview image, possibly via an interface to Science Pipeline. • Science Archive Requirements document was drafted and has been discussed by the SSR. • Many use cases for the ASA have been drafted and were given to the developers. (http://almasw.hq.eso.org/almasw/bin/view/Archive/AsaUseCasesList) • Support for storage and association of ‘advanced products’. • Catalogue Support: offer access to catalogues (calibrators, spectral lines) and calibration database (R4.0) • Consists of two major parts: • ASA portal: Main access point to all query and retrieval services. • ASA data repositories and ASA services. • VO aware by sharing technologies and standards. • ASA VO registry (planned) • ASA VO SkyNode (planned) ASAC Meeting- Computing IPT

13. ALMA Science Archive (ASA) • Exercise use cases for each user type: • ALMA staff, Archive Users/PI and Reduction Pipeline • Test interfaces (query structure, ease of use); transfer content (connections between APDM structures in ASA made & distributed correctly); & performance. • Test data transfer from Operations Archive and ASA. Operations Archive (Santiago) Dataset ALMA Science Archive Queries Query Responses Commands Archive Subsystem State Data Correlator Data Quality Data Processed Data Archive Users, PI, VO, Staff ALMA Staff Reduction Pipeline ASAC Meeting- Computing IPT

14. Add a user to the repository: Support for ASA users Staff & PIs:User repository development: manage database of registered users Manager view of users: User edits his/her contact information: ASAC Meeting- Computing IPT

15. Archive table browser prototype list of ALMA Science Data Model (ASDM) tables entities table populated by   Correlator(actual contents not shown) ASAC Meeting- Computing IPT

16. Observing Preparation • Support creation of Phase 1 proposals and Phase 2 programs, with submission through the ALMA archive. • “APDM” refactored (R4) • Visual Spatial Editor – first version. Visual Spectral Editor also exists. Final versions of both for R6. • Support for Director’s time (R5) • “Special SBs” for fixed period scheduling now available. (R4) • Phase I “bureaucracy” and textual cases (R5). • Phase I – II transition (creation of Phase II from approved Phase I) (R5) • Support all ALMA Observing modes • So far optical pointing and holography observatory modes, and Single Field Interferometry (SFI) observing mode. • infrastructure in place to allow rapid development for further modes, R5-6 and on. • Provide a mode which operates in terms of science goals with automated generation of SBs. • Science goal-based input for Phases I and II – First “wizard” now available for SFI. Will expand as modes are developed. • A generic framework for science input collection and SB generation is in place. • Alternative science goal presentation (“workbooks” - more flexible for users than a wizard) under development – R4. • Almost complete infrastructure will allow science goal input and SB generation to be extended to more observing modes. R5-6 and on. ASAC Meeting- Computing IPT

17. Observing Preparation (cont.) • Exposure time and data rate calculators shall be provided. • ETC complete – in maintenance. Supports all arrays. • Data rate estimator first version available, requires more user friendly presentation (R5.1) • Execution likelihood estimator (R5) • Sensible default values and basic validity checking will be provided. • Infrastructure to support validation and setting of (sensible) defaults is in place. • Needs to be rolled out for all items (R4.1) • Provide an expert mode with access to technical parameters of interest to observers (e.g., detailed control of correlator configuration). • Almost complete for main array – in maintenance as other specs emerge. Still details to be determined for ACA/TP arrays. (R5) • Local disk persistency (provided by MicroArchive). In maintenance. • Source list and catalog (target and spectral) support • Infrastructure in place with interim catalogs. • To be moved to Observatory Characteristics (R4) • Final catalogs in place by agreement with Operations (R6?) • Widely portable software. • Supported now, will continue. ASAC Meeting- Computing IPT

18. Spatial Visual Editor ASAC Meeting- Computing IPT

19. Observatory Operations Software (ObOps) Passed its PDR at CDR4 (June 2006). Still at .5 FTE/yr effort • Infrastructure: Ops Support Database, Data Access Module, etc. (R4-R9) • Shift Log Tool: record all day- and night-time observing activities • core features (R4) • basic features (R5) • full implementation (R6) • Observatory Characteristics Database GUI (R4.1) • Phase One Manager: support for proposal review process (R6) • ProjectTracker: control and monitor operations information flow (R6) • User Portal: integrated access to operations info • internal (ALMA only) access (R7) • public access (R9) • QualityTool: data quality assurance (R7) • DataPacker • raw data delivery (R7) • data package production and delivery (R9) • PublicationsDatabase (R9) ASAC Meeting- Computing IPT

20. Offline software • Priority 1 requirements. In particular for the next release: • Single dish analysis, High dynamic range imaging (R4.0) • Standard calibration (R4.0) • CASA framework doc. • Portable to a range of Unix-like operating systems. • Efficient enough in operations to satisfy user expectations and for use with the Science Pipeline. • GUI ,CLI, and script interfaces (Image viewer & data visualization GUIs available) • Readily installed by end users for processing execution. GUI tools for raw data and image inspection, flagging, arbitrary plotting. • Filler for ALMA Export Data Format for Interferometric data (single dish ASDM filler to be developed when ASDM format defined). • ASDM extensions, EDF doc (R4.0) • Extensive support for multi-channel, multi-polarization, multi-spectral window, multi-pointing flagging, calibration, imaging, and analysis for interferometric, single dish, and combined data from existing telescopes. • Simulation capability. • Processing history, interactive warnings, and re-execution facilities. • Direct access to the ALMA Archive and the VO. • Atmospheric modeling for the ALMA bands • Extensive user documentation ASAC Meeting- Computing IPT

21. AIPS++ Glish interface; unknown, unsupported outside of NRAO Tasking system based on Glish GUI system based on Glish/Tk; limited widgets, not robust! Difficult for external developers to contribute Multi-CD binary distribution Large monolithic libraries with cross dependencies No namespace Freeze 2006.75 CASA Python interface (community standard); IPython extensions Direct binding to Python; ACS, other frameworks readily possible Hierarchical set of small libraries with clearly defined dependencies Namespace protection for integration with other code RPM distribution mechanism; auto-updates possible Robust Inherits all application code improvements in robustness and performance. CASA ASAC Meeting- Computing IPT

22. DataCapture • Maps telescope, observing information into Alma Science Data Model (ASDM) for post-processing in near-real time. ASAC Meeting- Computing IPT

23. 2006 NRAO: NAUG testing (AIPS++) Community: Project tests ALMA: external science testing: +single-baseline commissioning test) (CASA) ALMA: external science testing: +single dish reduction (CASA) ALMA: commissioning support (CASA) ALMA/EVLA: user interface review (CASA) EVLA: external science testing: +full polarization imaging; antenna pointing calibration 2006.5 AIPS++ frozen 2007 NRAO: User support (CASA) ALMA: commissioning support (CASA) EVLA: external science testing: + wide band calibration imaging (CASA) ALMA: P1 SSRs complete 2008 EVLA: external science testing: RFI (CASA) NRAO: user support (CASA) ALMA: commissioning support EVLA: P1 SSRs complete 2009-2010 ALMA/EVLA: commissioning support Community: CASA released/distributed for early ALMA/EVLA science 2011 Community: CASA released/distributed; full user support (ALMA/EVLA: P1/P2s) CASA Timeline ASAC Meeting- Computing IPT

24. Offline User Test 4 (report Apr.06) Test delayed by SSR (not developers) so the first framework changes and python commands could be evaluated in a realistic environment of single baseline interferometry. CASA is a python binding of the C++ functionality that was in AIPS++. This initial version of CASA had python commands that had a one-to-one correspondence with AIPS++ Glish commands. • first test of the CASA infrastructure and python interface. • focus was to exercise and verify functionality for • editing, visualization, data reduction and analysis for single baseline interferometry – • capabilities that are needed at the ALMA Test Facility and for commissioning support. • Testers with extensive experience commissioning radio telescopes and/or operating telescopes. • In general, single-baseline data reduction procedures worked with good stability and accuracy. • Additional visualization capabilities to support commissioning activities still needed. ASAC Meeting- Computing IPT

25. Increased capability Offline Development & Test Schedule Community use, command line python Limited release, mostly command line, some GUIs Single dish reduction & imaging Python tasks, new framework Delay: late arrival of ACA requirements. User Focus group Tools  Tasks Single-BL Commissioning at ATF 1st Python interface, new framework Delay: SSR decision to allow testing of 1st python interface. Interferometer/TP combo Complete VLA+GBT & BIMA+12m Small Multi-field Mosaic VLA & BIMA mosaics User Acceptance Tests Single Field VLA & PdBI 13mm 1/3mm 2003 2004 2005 2006 2007 ASAC Meeting- Computing IPT

26. Offline Key Planned Development for 2006/2007: • Commissioning Plan shows heavy use of the offline package. • Change development priorities/details (e.g.: single baseline analysis software must be ready earlier than expected; more plotting capabilities needed). • Completed commissioning support test in February 2006 (1st step before using ATF) • Single-dish reduction & imaging – will be based on ASAP package • Single-dish user test scheduled for late 2006 (dependent on ACA requirements). • Also evaluate task-based reduction & imaging for interferometric data. • Complete ALMA interferometric ASDM filler and new viewer (more robust). Formal tests have been limited to specific datasets to verify development along a defined path. This has been necessary for development but need to expand to more open use once python command line interface is stable – 2007 – 2009: Staged plan to release CASA to astronomers for scientific use. Not final yet, however, key components are: • Nov 2006: CASA user point person starts (guide user interface development, documentation & help new users) • 2007-2008: Increase number of NRAO scientists & a few external scientists reduce own data • Mid 2007: Early release of CASA simulator • Mid-2008: Release to the public. ASAC Meeting- Computing IPT

27. Science Pipeline Goals Automated flagging, calibration and imaging with associated quality information for common ALMA observing modes. Pipeline processing shall be data driven). Main purpose is to produce: calibrated, scientifically useful images. • Following information will be calculated or available for use in pipeline processing: sideband ratios, temperature scale from an atmospheric model, bandpass calibration, instrumental polarization, phase rms, antenna efficiencies from source with known flux, and time variable amplitude and phase curves. (heuristics computing bandpass, gain, and flux calibrations for single field interferometry data, (R3.1->) • Choose: whether or not to use WVR corrected if both are available, to reuse previous calibrations if necessary, to select appropriate deconvolution algorithm, to do self-calibration or not, to do continuum subtraction or not (Choices non-existent or hardcoded, (R4.1->). • Automatically assemble all needed information from the Archive (i.e., assemble appropriate sources from multiple SBs over multiple epochs / configurations). (automatic ASDM retrieval for observing modes with simple observing unit set structures supported, (R2.1->) • User friendly processing summary and processing script which can be modified and used in later reprocessing. (processing summary logs are written by current pipeline heuristics scripts, automatic script logging may be available from CASAPY, (R4.0->) • Scripting interface to allow operations to modify processing heuristics without changes in compiled code. (Scripts will be written in Python, (R4.0) • Manual mode (e.g., for reprocessing particular projects). (R5.0->) • First version of pipeline processing scripts. (E2E for: single field interferometry (R4.0->), for: single dish: (R4.1->) ASAC Meeting- Computing IPT

28. Status: Heuristics • Single field interferometry (prototype e2e heuristics) • calibrator source baseline based flagging heuristics • bandpass, gain, flux calibration solution heuristics • calibrator source imaging heuristics • some quality assessment measures • calibrator source antenna based flagging heuristics • target source flagging and imaging heuristics • Single dish (baseline fitting heuristics) • including line detection and masking heuristics • Software • general Python classes for flagging, calibrating, and imaging SFI data • general Python classes for baseline fitting (some cleanup required) • Heuristics development slower than expected (based on 2004 planning) • Reasons: Technical: CASA framework conversion happening at same time; • Developmental: more difficult to get flagging heuristics correct than expected; • Personnel: ramp up to expertise needed to develop interferometric heuristics slow. ASAC Meeting- Computing IPT

29. Status Pipeline Infrastructure • Quicklook • supports parallel observing sessions (independent subarrays) • provides clean separation of processing and display functions • working telescope calibration results monitor and GUI • Science • working heuristics development framework (IPython, CASAPY) • working set of data processing engines (CASA, ASAP) • heuristics not yet connected to infrastructure • ASDM filler prototype • separate ALMA / CASA build environments Modules • full set of tool methods available to heuristics from CASAPY for • table, ms, constants, quanta, autoflag, calibrator, and imager • subset of tool methods available to heuristics from CASAPY for • image, measures • prototype visibility display tool already available to heuristics from CASAPY • msplot • Java based table browser available in latest stable • to be integrated with CASAPY ASAC Meeting- Computing IPT

30. Heuristics User Test 3 – Browser Snapshots PdBI Spectral line From R3 User test conclusions (mar.’06): The quality of the automatic flagging was judged to be good by the majority of the testers and the calibration heuristics were found to be a reasonable first implementation ASAC Meeting- Computing IPT

31. Pipeline - What is ahead for next year ? • Science • E2E single field interferometry heuristics running in the ALMA environment • E2E single dish heuristics • prototype cluster computing environment • QuickLook • user friendly telescope calibration monitor and display • real time visibility display using real time filler • some bad data detection heuristics integrated with alarm system ASAC Meeting- Computing IPT

32. Dynamic Scheduling Software to automatically choose best SB given scientific priority, current environmental conditions, current and future configurations, and the total pool of possible observations • Complete interactive - and queue - scheduling (R4.0) (R4.0). • Planning mode simulator (incl. weather data and quality - R4.0) (R4.0) • SB interruption and re-execution. (Special SBs for fixed-time scheduling - R4.0) (R5-6) • Manual override of automated choices. (R4.1) • Mechanisms for operations to change the weight of factors within the dynamic scheduler without significant programming. (R4.1) • Scheduler reporting facilities. (GUI for master scheduler, Exec plug-in - R4.1) (R4.1) • Heuristics delivered by CIPT will require optimization by operations. (R5) • Multiple sub-arrays supported (including taking antennas off-,on-line - R4.1) (R4.1 ) • ACA projects operating independently (R5-6) • Medium term scheduling (R5-6) • Long term scheduling (R7-8) • Receiver setup optimizations (R7-9) • [Breakpoint facility – low priority.] (R7-8) • Accounting of time spent for various purposes (source, programs, observatory calibrations). (R?) ASAC Meeting- Computing IPT

33. Operator Master Client Scheduler ASAC Meeting- Computing IPT

34. TelCal Online (and Offline) Calibrations Telescope Calibration takes scans (mainly special calibration scans) and performs data reduction on them. Results are used by Control for further observations with improved parameters and by the Quick Look Pipeline to display and calibrate quick look data. Some array calibrations are not required to be performed on-line (e.g. Holography, pointing models). • Holography (Pointing & focus offset, illumination pattern, surface err.) (R4.0) • Atmospheric calibration (Temperature scale) (R4) • WVR values (R5) • Sideband ratios (R4) • Phase calibration (phase rms, calibration curves) (R4) • Amplitude calibration (calibration curve) (R4) • Bandbass calibration (R5) • Reference pointing (publish/reduce scan results) (done) • Focus (partly done) • Delay (publish/reduce delay scans) (R4.0) • Pointing Model (optical done) • Baseline measurement (positions from delays) (R4.0) • Skydip • Interferometry solvers (done) • Automatic activation (according to intent) (R4.0) ASAC Meeting- Computing IPT