410 likes | 506 Views
CyberInfrastructure (CI) Build. Review of the Development Process. Concept of Operations (CONOPS) Describes how system should function at a high level Use Cases Flows from CONOPS, describes how users interact with system Requirements Development
E N D
Review of the Development Process • Concept of Operations (CONOPS) • Describes how system should function at a high level • Use Cases • Flows from CONOPS, describes how users interact with system • Requirements Development • Derived from Use Cases and CONOPS; list of “Shalls” that describe the functionality of the system • Software Development • Implementation of requirements into software • Software Testing • Verification that the software performs as specified by requirements • Validation • Validation that the system performs a desired by the user, and provides valid output data products
CI Requirements Development • Requirements Development Path • Capability list from the uFrame architecture • Based on other uFrame instantiations (such as Tsunami, AWIPS II, etc.) • “Must have” list • List derived from use cases and CONOPS • Missing “must haves” added to CI Capability list • Vet the updated CI Capability list through the CCB to gain concurrence from all stakeholders • Follow-on Tasks • Omaha team developing functionality based on capability list • Rutgers team developing test procedures based on capability list
Software Development Teams • Instrument Drivers, Dataset Agent Drivers, Algorithms • Software interface to sensors, data processing • Drivers: Raytheon IDS & UW software engineers • Algorithms: OSU • uFrame (CI) Functionality • Core Infrastructure: data ingestion, storage, web services • Raytheon IIS (AWIPS Team) • User Interface Functionality • User Interface screens, user functionality • ASA Science
System Summary UI Drivers/Alg.
Prior Build Deliveries • Build 1 Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database • Build 2A Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links) • Build 2B Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI • Build 3 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status • Build 4 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status
Build 5 Composition • 10 Drivers(OL: Dan Mergens) • 8Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities(Omaha: Scott Risch) • Mission Execution • Quality Control Algorithms execution/results display • Non-Time Series Plotting • Asset Management Updates • Alarm Notification • UI Capabilities(ASA: Dan Maher) • Final User Management • CI Status Metrics
Build 6Composition • Drivers(OL: Dan Mergens) • 21Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • Data Annotation • Telemetry Status (feeds into Health Status) • DOI Schema support • Further Metadata Access/Retrieval • Updates/Fixes • Bug fixes will be worked by priority • UI Capabilities (ASA: Dan Maher) • Updates/Fixes
Driver StatusSummary • Driver Development Stages: • IDD, Code/Unit Test, Software Integration Testing • Summary of Driver Development Progress • Uncabled: 181 Dataset Agent Drivers / 3 remaining • 1 (camds_abc_dcl) in work • 2 (flord_g_dcl, zplsg_a_sio_mule) – no test data available • Cabled: 32 Instrument Agent Drivers / 1 remaining • Driver Integration Stages • uFrame OOI Integration, Verification, Validation
uFrame OOI • uFrame CI focused on: • Leveraging existing Raytheon uFrame SOA technology • Improved User Interface • Simplified hardware and network architecture • Collocated Data Management organization • Incremental Build Schedule • Allows for identification and resolution of integration issues • Permits feedback into later builds • Systems Engineering Management Plan (SEMP) • OOINet Build schedule is linked with new SEMP, which defines commissioning plan, specific deliverables, and is represented in schedule
uFrame OOI Architecture Notes • Plug-in Development • The uFrameteam is developing or extending plug-ins to meet some of OOI CI capabilities. • Other capabilities will be implemented via hardware or network infrastructure and via graphical user interface development by other OOI CI development teams. • Database • Postgress database implemented to hold metadata about ingested data files and to hold the information about the physical assets that collect the data. • Streaming Ingest • Data ingest performed using Cassandra engine. Cassandra provides for better performance than original (PyPies) technology, and includes support for clustering (allowing scalability), and replication.
Data Ingestion Processing • Profilers & Moorings • Input through Java Messages to Apache Qpid queue • Streaming Data • Instrument Agent driver translates data stream into a Qpid queue Streaming • Data Streams ingested via Sensor Integration extension • Python parsers used to parse incoming data • L0 Data stored within Cassandra • Meta data stored within Postgress
Algorithm Processing • Input Data streams (L0) from Mooring/ Node/ Sensors are transformed to output Product Derived (L1 & L2) via algorithms • Algorithms require that calibration information be stored and managed (as calibration data may change over time Data arriving via streams is converted to usable data products through application of algorithms which apply calibration constants, perform quality control functions, and merge data from co-located sensors to generate data products
Data Management/Processing • Asset data is retrieved from OOINetuFrame via REST services hosted by the Web Services plug-in to uFrame. • The REST request from the Asset Management GUI is handled by the Asset Web Service which translates the REST request to a database query. Asset Data can be ingested by bulk upload (for operators) and via Web based GUI GUI will provide for entering individual data records, including editing and viewing existing records
User Interface Functionality • UI Capabilities will be available with Build 5 Delivery • Some UI Capabilities dependent upon uFrame capabilities • Integration testing to continue with Build 6 • Expect some changes from user feedback load testing
User Interface Functionality • Screens can be coded and unit tested, but not fully tested until full System is functional
OOI-CI Testing • Driver Development Unit / Integration Test • Verify that drivers function per IDD • Performed by Driver Development Teams (Raytheon, OSU, UW) • Framework Development Unit / Integration Test • Verify that software functions as expected in the development environment • Performed by Raytheon Omaha team • Verification / Acceptance Test • Verify that software supports defined use cases and meets documented requirements • This is done by running through test cases which are created from use cases • Perform by Rutgers Software Test team • Data Validation Test • Verify that data transformation algorithm is correct via output data validation • Perform by Rutgers Data Management team • Science Test • Verify that software meets science needs • Perform by wider science community
Test Artifacts Test Procedures Test Reports Test Planning Test Scheduling
User Interface Demonstration Video of walkthrough of UI screens
Back-up Slides Detail slides provided as back-up data follow this slide
Instrument Driver Overall Status Software Development Effort Systems Integration and Test Effort
Dataset Parser Plug-in Progress Software Development Effort Source: Unified Status Tracking - DAD Status
Build 1B Composition • Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database
Build 2A Composition • Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links)
Build 2B Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI
Build 3 Composition • Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status
Build 4 Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers • 15 Instrument Agent Drivers • 6 Platform Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Alerts/Alarms/Notifications • User specified Annotations • UI Capabilities (ASA: Dan Maher) • Initial Command/Control functionality • QA/QC functionality; Alerts & Alarms • Help/Glossary/How-to • Initial Asset Management
Data Management Science/Engineering Data Data Management Cabled/Platforms Instruments L0 Data Uframe Data Ingestion Parser to data stream association Drivers/Parsers Asset Management: eg. Location, Depth, Calibration Coefficients, startup config, telemetry Creation of L1, L2 Data Products Command and Control QC look up tables Algorithms Mission Execution QC on L1, L2 Data Products Alarm, Alert, Health/Status flags and thresholds Glider Schedules Alerts/Alarms, Health Status monitoring Profiler Mission Plans
Asset Management Database Schema for storage of all Metadata related to assets and their associated events
Earned Value Definitions • CPI = Cost Performance Index • Work Performed/Cost of work Performed, 1.0 = Spending as planned • SPI = Schedule Performance Index • Work Performed/Work Scheduled, 1.0 = Completing work as planned • Cur. = Current Period Performance • Status of most current 2 weeks of work • Cum. = Cumulative Performance • Status of work since beginning of task • Plot = Historical data • Plot is bi-weekly graph of cumulative Plan Costs/Actual Costs/Performance Costs
UI Plan vs. Actuals Showing data through end of Apr. Plot is updated at the end of the month, per ASA financial report –