EVLA Software - Overall Architecture, Science & Online Domains

EVLA Software - Overall Architecture, Science & Online Domains Bryan Butler NRAO

Observation Preparation EVLA VLBA ALMA GBT VLBA Sched GBT Sched ALMA Sched GBT Control VLBA Control ALMA Control NRAO-wide Design Mostly Telescope-Independent Common Software Proposal Submission And Handling Observer Domain Mostly Telescope-Specific Project Software EVLA Sched Telescope Domain EVLA Control Telescope Data Model GBT Postproc Feedback to telescope Data Capture Science Data Model Telcal Quick Look Mostly Telescope-Independent Common Software Science Domain Archive Pipeline Export Data Format Offline VO EVLA Advisory Committee Meeting

NRAO-wide Design The EVLA software system needs to conform to the NRAO-wide design to the extent that it has been developed. Unfortunately, the NRAO-wide effort has languished for the past two years (this is changing - a new “e2e Project Manager” [Nicole Radziwill] and “e2e Project Scientist” [Ed Fomalont] have been hired in CV). EVLA Advisory Committee Meeting

Domains & Models The NRAO-wide design introduced the concept of “domains” - different (large) pieces of the system that can be grouped sensibly. It presented three such domains: • Observer • Telescope • Science It also presented a number of “models” - descriptions of different (smaller) pieces of the system: • Observatory • Project • Observing • Archive • Telescope Data • Science Data EVLA Advisory Committee Meeting

EVLA Overall Design In the spring of 2004, the EVLA software group undertook an effort to develop and document a high level design (led by Morgan, Ryan, Sowinski, and Waters). This culminated in a completed design, which was reviewed by the NRAO eOC in June 2004. The design presented in the next few slides is based mostly on that effort, with extension and modification in the past two years. EVLA Advisory Committee Meeting

EVLA Software Requirements The software design and implementation is driven by a number of requirements documents: • e2e Science Software Requirements • Engineering Software Requirements • Real-time System Software Requirements • Operations Software Requirements These do not have everything in them (for instance Proposal Handling and User Database, which are covered in separate [less “requirementy”] documents), but are fairly complete, and notably, the e2e requirements document has extensive requirements for PST and OPT. EVLA Advisory Committee Meeting

Major Elements (“Models”) The main flow of information (and processes; the “workflow” or “dataflow”) is: A Scheduling Block is an atomic unit of observing. It is made up of a sequence ofscans; a scan is made up of source(s), resource(s) (hardware definition - both FE and BE), timinginformation, and a “mode”. The mode defines the subscan(s), which are comprised of a single source, resource, and timing information. Proposal Data Project(s) Commands Program(s) Schedule(s) EVLA Advisory Committee Meeting

EVLA High Level Architecture DATAFLOW EVLA Advisory Committee Meeting

EVLA High Level Architecture Note that most of the major subsystems have a direct counterpart in current VLA software - we have a significant amount of experience in what is needed there. What has been lacking is the electronic storage and passage of information between subsystems, and therefore the ability to do much of this automatically. EVLA Advisory Committee Meeting

EVLA Design Detail, Pre-Observing Science Domain Proposal Submission Tool (PST) Proposal Portal Proposal Handling Tool (PHT) Astronomer or Staff Authenticated Astronomer or Staff EVLA Observing Heuristics Project Observation Preparation Tool (OPT) Program Block (Set of Scheduling Blocks for one Program) To Observation Scheduling Tool EVLA Advisory Committee Meeting

EVLA Design Detail, Pre-Observing Online Domain From OPT Archive Observation Scheduling Tool (OST) Operator Environment Heuristics Metadata to DCAF Execution State Next SB Archive Executor Operator Metadata to DCAF Results from TelCal Equipment State Sequence of Configurations Antenna Delays From AMCS & CMCS To AMCS & CMCS EVLA Advisory Committee Meeting

EVLA Design Detail, Real-Time Domain From Executor State Counts FOTS Receiver Station, Baseline Boards EVLA Antennas RF CBE Lag Frames AMCS FF Raw Vis CMCS Hardware M&C Equipment State, Data Addressing Info, Messages, Alerts, etc. To Archive & TelCal To DCAF To DCAF EVLA Advisory Committee Meeting

EVLA Design Detail, Post-Observing Online Domain Astronomer or Operator To Executor And Archive From AMCS & CMCS From CMCS Portal TelCal Results Data Capture And Format (DCAF) TelCal Authenticated Astronomer or Operator SDM SDM M&C Archive Observation Status Monitor Tool (OSMT) Quick Look Pipeline (QLP) M&C Archive To Archive (?) To Archive EVLA Advisory Committee Meeting

Archive EVLA Design Detail, Post-Observing Science Domain Astronomer From DCAF From CMCS Open Products Archive Search and Retrieval Tool (ASRT) Portal Cubes (?) Existing Proprietary Products Image Cubes Open Products Post-Processing Default Image Pipeline (DIP) Trigger Authenticated Astronomer VO Astronomer Reprocessed Proprietary Products EVLA Advisory Committee Meeting

Detailed Subsystem Designs & Prototypes For most of the major subsystems, we either have a very advanced prototype (Portal, PST, Executor, OSMT), an early prototype (PHT, OPT, OST, ASRT, TelCal), or at least a much more detailed design (DCAF). The areas where we have done little work and in fact have only preliminary (high level) designs are for the pipelines (QLP, DIP). EVLA Advisory Committee Meeting

Portal, User Database, Authentication We know that we need some way to restrict access to the various tools, and the information available within them. We do this with a “portal”, through which users access the various tools. It authenticates users, and generates a unique token which is then used to verify a user’s login status. We have our own implementation of this, but recognize that we may need to migrate to the VO method (or have a translation layer). EVLA Advisory Committee Meeting

Portal, User Database, Authentication EVLA Advisory Committee Meeting

Portal, User Database, Authentication Users enter their own User Database information, except: • Institution information - they can only select an institution from a pre-set list (we want to use this to automatically generate reports to the NSF, which require precise information on institutions) - if the institute is not there, they indicate so, and we (well, I) add it. • We allow so-called “3rd party” user registrations during proposal preparation and submission, adding significant complexity (a sore spot with us, but demanded by operations). EVLA Advisory Committee Meeting

Proposal Submission Tool (PST) • Mainly a tool to collect form data (web browser) • Mostly telescope independent, with “resources” the exception • Used to enforce telescope “policy” • Coupled to other EVLA tools only loosely, via Portal and databases. • Currently also supports GBT. EVLA Advisory Committee Meeting

PST - Main Processes Main Processes: • Model - retrieve and write data to database • Controller - business logic to map user input (from browser) into objects which are then written to database • View - the look-and-feel of the interface (done in browser) • Validation of various fields - an important and significant part of the tool • Help system EVLA Advisory Committee Meeting

PST - Model The Model drives everything, so what’s in there? • science information - title, category, “mode”, abstract, scientific justification, and some misc. info. • Authors, including which is the PI and “contact author” • Sources • Resources (telescope hardware setup) • “Sessions” (a guide to SB setup) • Student Support Essentially, everything that is necessary to: • Referee the proposal • Assign telescope time (and money) • Automatically generate SBs (mostly for novice users, but experienced users will use this too!) EVLA Advisory Committee Meeting

PST - “The View” Although the logic is driven by the model, most of the programming complexity here is in the view. How do we do this? 6 main “tabs” in the browser, to represent the 6 main parts of the model. There is also some superstructure, to allow for higher level functions (edit/create, submit, print, choose telescope, etc.), but, again, most of the complexity is in the view for these 6 tabs. EVLA Advisory Committee Meeting

PST - “The View” Let’s look at the actual tool… EVLA Advisory Committee Meeting

PST - Submit We must support both normal “deadline” submissions, as well as “Rapid Response” submissions (this is fundamentally a policy issue). Upon submission, the proposal handling process is initiated. What is stored is a “proposal” (as represented by a Proposal Data Model). This is NOT the Project Data Model, but rather is a guide to the creation of the initial PDM. This is an important distinction, and something we came to a recent agreement on with ALMA (for them this is the Science View of the PDM). EVLA Advisory Committee Meeting

Proposal Handling Tool (PHT) • Presently, only very initial “wrangling” (view, print), and then splits into GB and VLA specific handling • GB and VLA separately support: • Adding additional data (edit XML by hand or script) • Fixing ‘problems’ • Pretty-printing, for referees and scheduling committee • Future: • Referee process • Scheduling committee details • Points of interest: • Requirements for VLA are in hand, but not in the form of the detailed requirements for other areas, rather more like a “User Story”. We need to transform this to real requirements, including the GB process (which have not been written down to our knowledge). • Does this go in the PST (with maybe part in the OPT) or as a separate tool? • The fundamental output from the PHT is Projects, as represented by a Project Data Model. EVLA Advisory Committee Meeting

Observation Preparation Tool (OPT) This is the process that takes a more generic view of a Project, and turns it into something that can actually be used to command the hardware of the telescope (and run ancillary tasks). The fundamental output of the OPT is Program Blocks (PBs) (remember that a PB is a collection of SBs, with some extra information - mostly “contingencies”) It needs to support 3 “levels” of user: • Novice (automatic generation of PBs for “standard modes”) • Intermediate (this is the tough one!) • Expert (allow for script level editing) EVLA Advisory Committee Meeting

OPT - commonality with PST Objects in common with PST: • Sources • Resources (hardware configuration) Things not in common: • Things not in OPT: • Front page stuff • Authors • Sessions (well, kind of - since they “represent” SBs) • Student Support • Things not in PST • Scan builder and organizer • PB organizer • Detailed correlator setup tool • Calibrator tool EVLA Advisory Committee Meeting

OPT - Detailed Design EVLA Advisory Committee Meeting

OPT - Detailed Design Modify PB EVLA Advisory Committee Meeting

OPT - Detailed Design Modify SB EVLA Advisory Committee Meeting

OPT - Current Status We currently have an early prototype of the GUI (duplicating the look-and-feel of the PST) in place which will authenticate users and has the most minimal PB functions (create, delete, etc.). Much of our work here, however, has been on the specification of the details of the objects (the “data models”) for the various parts of the system. We are starting here, knowing that many of the elements in the system will be needed here and will be common. These include the definitions of Proposal, Project, Program Block, and Scheduling Block, and as parts of that, Sources, Resources, and Scans. We start out with an XML document provided by a domain expert, and then turn that into objects. We are currently having detailed discussions with ALMA to attempt to have as much as common in these objects. It is clear that early parts of the process (Proposal) can be common, and that general concepts (major elements of SBs, for instance) can be common; it is not yet clear the level to which true commonality will be achieved. EVLA Advisory Committee Meeting

OPT - Plan Our plan is to have new major functionality available within the OPT roughly every 3 months. Major milestones are: • 30Apr06: framework with minimal functionality • 31Jul06: Add VLA calibrator database access, initial spectral setup • 31Oct06: Full calibrator setup, more observation setup • 31Jan07: VLA mostly supported. Some validation/PB creation • 30Apr07: Beginning prototype WIDAR support • 31Jul07: VLA fully supported; prototype WIDAR mostly supported • 31Oct07: Prototype WIDAR fully supported EVLA Advisory Committee Meeting

Observation Scheduling Tool (OST) We (well, Barry) have been conducting tests of dynamic scheduling with the VLA during the past 2 (3?) reconfigurations. Again, we consider these as prototypes for the final EVLA subsystem - giving us invaluable information on the practical aspects of dynamic scheduling of a many-element radio interferometer. EVLA Advisory Committee Meeting

OST - Block Diagram EVLA Advisory Committee Meeting

OST - Lessons Learned Here are the lessons learned - I need the full list from Barry. A few things I know: • ALMA system didn’t work well (as expected, per Allen Farris’ email) • System is inordinately fond of short SBs • Currently effort-intensive (but getting better) EVLA Advisory Committee Meeting

OST - Plan Here is the plan. I need to get with Barry on this, but my understanding is that he wants the VLA fully dynamically scheduled by the end of ‘06 (yes, he has indeed said that!). For the EVLA-specific part, we are assigning some effort beginning summer ‘06 to this. EVLA Advisory Committee Meeting

EVLA Software - Overall Architecture, Science & Online Domains