1 / 41

CyberInfrastructure (CI) Build

CyberInfrastructure (CI) Build. Review of the Development Process. Concept of Operations (CONOPS) Describes how system should function at a high level Use Cases Flows from CONOPS, describes how users interact with system Requirements Development

olwen
Download Presentation

CyberInfrastructure (CI) Build

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. CyberInfrastructure (CI) Build

  2. Review of the Development Process • Concept of Operations (CONOPS) • Describes how system should function at a high level • Use Cases • Flows from CONOPS, describes how users interact with system • Requirements Development • Derived from Use Cases and CONOPS; list of “Shalls” that describe the functionality of the system • Software Development • Implementation of requirements into software • Software Testing • Verification that the software performs as specified by requirements • Validation • Validation that the system performs a desired by the user, and provides valid output data products

  3. CI Requirements Development • Requirements Development Path • Capability list from the uFrame architecture • Based on other uFrame instantiations (such as Tsunami, AWIPS II, etc.) • “Must have” list • List derived from use cases and CONOPS • Missing “must haves” added to CI Capability list • Vet the updated CI Capability list through the CCB to gain concurrence from all stakeholders • Follow-on Tasks • Omaha team developing functionality based on capability list • Rutgers team developing test procedures based on capability list

  4. Software Development Teams • Instrument Drivers, Dataset Agent Drivers, Algorithms • Software interface to sensors, data processing • Drivers: Raytheon IDS & UW software engineers • Algorithms: OSU • uFrame (CI) Functionality • Core Infrastructure: data ingestion, storage, web services • Raytheon IIS (AWIPS Team) • User Interface Functionality • User Interface screens, user functionality • ASA Science

  5. System Summary UI Drivers/Alg.

  6. Prior Build Deliveries • Build 1 Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database • Build 2A Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links) • Build 2B Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI • Build 3 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status • Build 4 Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status

  7. Build 5 Composition • 10 Drivers(OL: Dan Mergens) • 8Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities(Omaha: Scott Risch) • Mission Execution • Quality Control Algorithms execution/results display • Non-Time Series Plotting • Asset Management Updates • Alarm Notification • UI Capabilities(ASA: Dan Maher) • Final User Management • CI Status Metrics

  8. Build 6Composition • Drivers(OL: Dan Mergens) • 21Dataset Agent Drivers • Includes telemetered and recovered variants • 2Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • Data Annotation • Telemetry Status (feeds into Health Status) • DOI Schema support • Further Metadata Access/Retrieval • Updates/Fixes • Bug fixes will be worked by priority • UI Capabilities (ASA: Dan Maher) • Updates/Fixes

  9. Driver StatusSummary • Driver Development Stages: • IDD, Code/Unit Test, Software Integration Testing • Summary of Driver Development Progress • Uncabled: 181 Dataset Agent Drivers / 3 remaining • 1 (camds_abc_dcl) in work • 2 (flord_g_dcl, zplsg_a_sio_mule) – no test data available • Cabled: 32 Instrument Agent Drivers / 1 remaining • Driver Integration Stages • uFrame OOI Integration, Verification, Validation

  10. uFrame OOI • uFrame CI focused on: • Leveraging existing Raytheon uFrame SOA technology • Improved User Interface • Simplified hardware and network architecture • Collocated Data Management organization • Incremental Build Schedule • Allows for identification and resolution of integration issues • Permits feedback into later builds • Systems Engineering Management Plan (SEMP) • OOINet Build schedule is linked with new SEMP, which defines commissioning plan, specific deliverables, and is represented in schedule

  11. uFrame OOI Architecture

  12. uFrame OOI Architecture Notes • Plug-in Development • The uFrameteam is developing or extending plug-ins to meet some of OOI CI capabilities. • Other capabilities will be implemented via hardware or network infrastructure and via graphical user interface development by other OOI CI development teams. • Database • Postgress database implemented to hold metadata about ingested data files and to hold the information about the physical assets that collect the data. • Streaming Ingest • Data ingest performed using Cassandra engine. Cassandra provides for better performance than original (PyPies) technology, and includes support for clustering (allowing scalability), and replication.

  13. Data Ingestion Processing • Profilers & Moorings • Input through Java Messages to Apache Qpid queue • Streaming Data • Instrument Agent driver translates data stream into a Qpid queue Streaming • Data Streams ingested via Sensor Integration extension • Python parsers used to parse incoming data • L0 Data stored within Cassandra • Meta data stored within Postgress

  14. Algorithm Processing • Input Data streams (L0) from Mooring/ Node/ Sensors are transformed to output Product Derived (L1 & L2) via algorithms • Algorithms require that calibration information be stored and managed (as calibration data may change over time Data arriving via streams is converted to usable data products through application of algorithms which apply calibration constants, perform quality control functions, and merge data from co-located sensors to generate data products

  15. Data Management/Processing • Asset data is retrieved from OOINetuFrame via REST services hosted by the Web Services plug-in to uFrame. • The REST request from the Asset Management GUI is handled by the Asset Web Service which translates the REST request to a database query. Asset Data can be ingested by bulk upload (for operators) and via Web based GUI GUI will provide for entering individual data records, including editing and viewing existing records

  16. User Interface Functionality • UI Capabilities will be available with Build 5 Delivery • Some UI Capabilities dependent upon uFrame capabilities • Integration testing to continue with Build 6 • Expect some changes from user feedback load testing

  17. User Interface Functionality • Screens can be coded and unit tested, but not fully tested until full System is functional

  18. User Interface Examples: Landing Page

  19. User Interface Examples: Infrastructure Info

  20. User Interface Examples: Science Data

  21. OOI-CI Testing • Driver Development Unit / Integration Test • Verify that drivers function per IDD • Performed by Driver Development Teams (Raytheon, OSU, UW) • Framework Development Unit / Integration Test • Verify that software functions as expected in the development environment • Performed by Raytheon Omaha team • Verification / Acceptance Test • Verify that software supports defined use cases and meets documented requirements • This is done by running through test cases which are created from use cases • Perform by Rutgers Software Test team • Data Validation Test • Verify that data transformation algorithm is correct via output data validation • Perform by Rutgers Data Management team • Science Test • Verify that software meets science needs • Perform by wider science community

  22. Test Artifacts Test Procedures Test Reports Test Planning Test Scheduling

  23. User Interface Demonstration Video of walkthrough of UI screens

  24. Back-up Slides Detail slides provided as back-up data follow this slide

  25. Instrument Driver Overall Status Software Development Effort Systems Integration and Test Effort

  26. Dataset Parser Plug-in Progress Software Development Effort Source: Unified Status Tracking - DAD Status

  27. uFrame Build Capabilities

  28. uFrame Build Capabilities

  29. uFrame Build Capabilities

  30. uFrame Build Capabilities

  31. uFrame Build Capabilities

  32. Build 1B Composition • Components: • Drivers (OL: Dan Mergens) • 6 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0 & L1 Data Products for the 6 Instrument Agent Drives • L1 Data Products • Web Services: Proof of Concept • UI Capabilities (ASA: Dan Maher) • None • Install/Deployment (Rutgers: Ivan Rodero) • Website • ERDDAP Database

  33. Build 2A Composition • Components: • Drivers (OL: Dan Mergens) • 23 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 3 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • L0, L1, L2 Data Products • Asset Management: • Record and manage mooring/platform data, expose UI interface • Web Services: • Test harness UI for plotting/charting, provide UI with access to data in Asset Management • UI Capabilities (ASA: Dan Maher) • Search & download science data • Graph science time-series data • Landing page (OOI Banner, navigational links)

  34. Build 2B Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers (each has a “telemetered” and “recovered” driver) • 0 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Asset Management: • Extended support for mooring/platform data that have not yet been deployed • Web Services: • Expose UI interface for query and data retrieval • UI Capabilites (ASA: Dan Maher) • Download support data • Query telemetered data from instruments • Landing page for Pioneer Array - Science UI

  35. Build 3 Composition • Components: • Drivers (OL: Dan Mergens) • 6 Dataset Agent Drivers • 4 Instrument Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Data/Asset Management: • Framework complete, including provenance tracking calibration and QC • Asset Management complete with full lifecycle management • Web Services: • Web framework complete: query/retrieval/application services • Integrate ASA user account management services • UI Capabilities (ASA: Dan Maher) • Manage User Accounts - Annotate Data • Update Metadata - Trouble Ticket Integration • Operator Turnover - Health/Status

  36. Build 4 Composition • Components: • Drivers (OL: Dan Mergens) • 14 Dataset Agent Drivers • 15 Instrument Agent Drivers • 6 Platform Agent Drivers • uFrame Capabilities (Omaha: Scott Risch) • All L0, L1, L2 Data Products for all drivers previously delivered • Alerts/Alarms/Notifications • User specified Annotations • UI Capabilities (ASA: Dan Maher) • Initial Command/Control functionality • QA/QC functionality; Alerts & Alarms • Help/Glossary/How-to • Initial Asset Management

  37. Data Management Science/Engineering Data Data Management Cabled/Platforms Instruments L0 Data Uframe Data Ingestion Parser to data stream association Drivers/Parsers Asset Management: eg. Location, Depth, Calibration Coefficients, startup config, telemetry Creation of L1, L2 Data Products Command and Control QC look up tables Algorithms Mission Execution QC on L1, L2 Data Products Alarm, Alert, Health/Status flags and thresholds Glider Schedules Alerts/Alarms, Health Status monitoring Profiler Mission Plans

  38. Asset Management Database Schema for storage of all Metadata related to assets and their associated events

  39. Earned Value Definitions • CPI = Cost Performance Index • Work Performed/Cost of work Performed, 1.0 = Spending as planned • SPI = Schedule Performance Index • Work Performed/Work Scheduled, 1.0 = Completing work as planned • Cur. = Current Period Performance • Status of most current 2 weeks of work • Cum. = Cumulative Performance • Status of work since beginning of task • Plot = Historical data • Plot is bi-weekly graph of cumulative Plan Costs/Actual Costs/Performance Costs

  40. uFrame CI Plan vs. Actuals

  41. UI Plan vs. Actuals Showing data through end of Apr. Plot is updated at the end of the month, per ASA financial report –

More Related