8:15 AM Tuesday September 15, 2009 Karl Frank, Point of Contact for Constellation Projects

1. Establishing IV&V Expectations 8:15 AM Tuesday September 15, 2009 Karl Frank, Point of Contact for Constellation Projects Diagrams for illustrative purposes only.

2. Section Overview • Goals in workshop context • Expectations: whose? With respect to what? • Goal-based capture • Role of Validation in model-based process • Model as cage for captured understanding • Model as share point • Role of Verification • Lessons learned

3. Section 1 In context of the workshop, this presentation aims at providing an overview of the many ways in which modeling has been done in the Fairmont IV&V facility, both how and why, so that you will be able to fit particular topics at a more detailed level, into a big picture. If successful, you will later be able to see how the use some particular diagram or tool, is part of an overall approach, not a detached effort. Goals in workshop context

4. Basis for Understanding • Modeling serves many purposes • Not an end in itself, serves V & V • Can be done in many different ways • Different “ways” as different languages • Within a given language different “ways”: • Levels of detail, “analysis” vs. “implementation” • Here, language is UML 2 • Levels of detail evolve to finer granularity • But never for purpose of implementation • Validation itself moves through “Levels”

5. Section 2 Collectively staff of an IV&V facility needs shared understanding of the goals and approach, and staff on a given project needs a shared understanding of that project, starting with project goals, and at the level of artifacts presented for validation and verification. Expectations: whose? WRT ?

6. Expectations • expectations in black box testing, is particularly focused on how the software under test should behave, not how it should have been implemented. • Behavior model takes precedence • Architectures & Interfaces require V&V • Upon presentation of an IRD what expectations do we bring to its analysis?

7. Section 3 Valid expectations wrt project artifacts begin with project goals. These are form the foundation of our project models. As there is a commonality to all NASA projects, the common goals of space transportation systems form a common root for all project models. Goal-based capture

8. Goal-Based Capture • Popularly called SysGoals Model • Adapted from use case modeling • Text document with use-case like structure • Graphic overview as UML use case diagram • System goals are elaborated as use cases • Use Cases bridge into the UML model • Head of the model navigation traces • Use case flow of events elaborated in Activity Diagrams, one per use case • UML model can be traced back to Goals

9. Common Model Pattern Diagram Example Goals at head of Navigation Links down to Activity Diagrams. Number of Levels will vary with the Project Horizontal Links Between alternative Views of same Behavior Swimlanes typically disappear at “functional” levels of behavior

10. Diagram Example One Project Use Cases Linked to Goals Diagram Example

11. Diagram Example, Different Project, Shared Goals

12. Section 4 Modeling process is tightly coupled with validation in steps. Goal statements from project sponsors or used to validate Concept of Operations documents from NASA and these are the sources that are the basis of the first iteration of the SRM, then used in validating next round of project requirements docs and specs. ROLE of validation

13. Validation • Project process presents successive levels of documents for validation • Approach focused on validating behaviors, modeled from validated sources, instead of text document focus, reporting on behaviors • Progress from System ConOps to sub-system, component, element level requirements, Interface Requirement Docs

14. Elaborated Behavior Example

15. Notation Example for Interface Validation Ports Discrete RIU remote interface unit

16. Section 5 For each project, a UML Model links goal and use case documents to behaviors and, thru behaviors, to source documents (Concept of Operations), architectures, issues raised in validation, and defects reported in verification. Do not think of Model as collection of diagrams: the traces to correlated requirements cannot be graphic. Progress on different IV&V tasks are traced thru model. Model as cage for captured

17. Cage for Captured Understanding • Diagrams as a training resource • Model as a shared database • Can report links to target requirements related to behaviors or other model elements • Same model elements can be viewed at different levels of detail for high-level or detailed views.

18. Section 6 Heading Different project tasks are best served by different kinds of diagram, based on different modeling concepts, and at different levels of detail. These are developed as views of a single ever-expanding underlying model, which can be audited programmatically for consistency, so that instead of producing a variety of distinct documents with diagrams, which need to be versioned as distinct resources, each project has a single shared and evolving UML model, organized into packages and subpackages, containing the gold copies of all the diagrams. Model as share point

19. UML Model database • Linked to Goal and Use Case Documents • Linked to Requirements Documents • Linked to Issues • Linked to exported Assertions • Linked to tests and test result. • The behaviors and logical architectures are the IV&V focus, and the model defines those, and links them to the other workproducts

20. Model as Share Point Correlation between expectations on behaviors -> requirements

21. Section 7 Heading Software verification requires models sufficiently detailed to make it possible to compare the model with the behavior or architecture of software. Our goals-based approach to model elaboration means that the needs of verification are met by the models only when they reach an advanced state of elaboration. In particular, statemachine diagrams are particularly useful in verification of software behavior, but for reasons to be explained, are among the last products of our modeling work. Role of verification

22. Verification • Statemachines in UML 2 made sufficiently detailed to be used as definitions of required behavior and prohibited behavior. • Oracle concept various external tools can interpret these statemachines as executable, and compare the behavior of software under test, to behaviors mandated or prohibited by the statemachine: assertion testing.

23. Statemachines

24. Diagram Example: Statemachine exportable Assertion for verification

25. Section 8 Ongoingchallenges: Many different UML modeling concepts and notations compete as ways to represent system behaviors and architectures. Up front agreement on what modeling concepts and graphic diagram standards is difficult to get, but important. Have modeling standards and model-based IV&V processes place. Lessons learned

26. Lessons • Distinction between Model, as a single organized XML database of model elements, and Diagrams which present graphic views of model contents, is difficult to master. • Many different UML modeling concepts and notations compete as ways to represent system behaviors and architectures. Up front agreement on what diagrams to use for what is difficult to get, but important. • Defining the process for applying concepts of model driven development to IV&V, while continuing to perform traditional IV&V functions. • Training modeling staff in validation and verification, and training validation staff in modeling, is a challenge.

27. Related Article • Latest issue IEEE Computer • Cover Feature seems to be based on approach NASA IV&V has been pioneering for years • Faultless Systems: Yes We Can! • Jean-Raymond Abriel, Swiss Federal Institute of Technology, Zurich Beware of arbitrary verbal distinction “horizontal” vs.“vertical” refinement