Data-Driven IV&V Decision Support

1. Data-Driven IV&V Decision Support NASA IV&V Software Assurance Tools (SWAT)

2. Outline • What is Decision Support? • Data Availability (On-Demand, Linked, Warehouse) • Data Presentation • Analysis, Reporting and Metrics • Related IV&V Efforts • Questions

3. What is Decision Support? • For our purposes, Decision Support will be defined as helping people make decisions that are supported by relevant data • A decision is defined as the choice of one among a number of alternatives • Decision making refers to the overall process of making a choice, which includes assessing available data and identifying alternatives

4. Data Availability Data Availability • Data must first be made available for decision support On-Demand Data Linked Data Data Warehouse

5. On-Demand Data • IV&V can support daily decision making with basic on-demand data access • Example #1 – Project PM may ask “Based on the issues we have identified so far, where should we focus IV&V resources?” • ORBIT (Integrity) Queries configured to show current project IV&V Issues by category and severity • Example #2 – Project PM may ask “Which of my team members should I allocate to an upcoming milestone review?” • JIRA Dashboard view configured to show current IV&V project tasking status by assignee

6. On-Demand Data • Example #3 – IVVO Lead may ask “Which project risks currently require greater attention?” • RiskManager 5x5 matrix view of risks filtered for the IV&V Office level

7. Linked Data • Linked Data for our purposes is defined as structured data from multiple sources that has been related in some way which can make other useful knowledge available

8. Linked Data • Each data entity is given an individual identity for access, e.g. a URL can be used if accessing through the web • The advantage comes from leveraging the relationships between data, not just the individual content itself • Linked data concepts vs. separate “silos” of data in spreadsheets for example provides the following benefits: • Data remains in an appropriate context, as it is referenced through a common interface (e.g. web service or centralized data store ) • Data is linked (related), enabling information to be combined across the organization or even from outside the organization • Data can still be accessible from other applications that depend on live data, while not preventing static data dumps where applicable (e.g. export to operational data store)

9. Linked Data • Vocabulary • Defining a formal and shared vocabulary is necessary to document relationships (ontology) • Permits “inferences” to be made when analyzing data • Search • Some value can be gained from just traversing linked data • Greater value is provided by some search (query) capability of linked data

10. Linked Data Data linked or related by project (or mission) IVVO Lead: “What types of IV&V issues or risks have we been identifying for missions with similar characteristics?” SQL Server Report generated based on data that is extracted from multiple data sources (e.g. IV&V issue tracking system, RiskManager tool, IV&V Projects/Mission database) into a centralized operational data store

11. Linked Data IV&V Project PM: “Which of my team resources may be overloaded with active issue follow up and additional tasking?” Data linked or related by analyst Dashboard display of team assignments for active IV&V issues (ORBIT) and other tasking (JIRA)

12. Data Warehouse • Historical analysis and trending must be based on data that is not just a single snapshot in time, rather data this is accumulated over a time period • Robust and strategic IV&V decision making can be supported through a data warehouse approach that adds a time (historical) dimension to the data

13. Data Warehouse

14. Data Warehouse TQ&E Lead: “Is there any correlation between IV&V issue generation rates and the maturity of methods employed by IV&V teams?” Data is analyzed over a period of time Business Intelligence (BI) tool configured to aggregate the necessary facts for analysis of any relationship Senior Management: “Do projects with senior level analysts generate more issues/higher severity issues?”

15. Data Presentation • Data must also be presented for decision support • Data can be presented in any number of ways for analysis: • Dashboard views of data in simple table and chart format, including custom interfaces that access other data sources to populate these views • SQL Server Reports delivered through a web interface • Excel can be populated from outside data to leverage additional analysis capability • Visualized data can be presented through Business Intelligence (BI) dashboards to help expose data patterns, trends and correlations

16. Analysis, Reporting and Metrics • Online Analytical Processing (OLAP) – multidimensional data and aggregations for analysis • Ad hoc reporting provided through such mechanisms as SQL Server Reporting Services (SSRS), with reports provided in web interface and e-mail subscriptions • Metrics (e.g. IV&V method utilization over time by projects) • Export to more lightweight options like Excel for specific analysis

17. Related IV&V Efforts • IVVO Data Management – related FY14 initiatives identified • Analysis Management Framework (AMF) domain model for IV&V analysis execution • Development of dashboards for SWAT developed tools that pull from various data sources • RiskManager and COMPASS tool web services • Atlassian gadgets for JIRA and Confluence

18. Questions ? ? Contact us at: ivv-swat@lists.nasa.gov