From Validating Models to Validating Systems

1. From Validating Models to Validating Systems Peter Denno 2013-02-25 University of Maryland ISR Colloquium

2. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work

3. Goals • Describe a “design philosophy” for systems that assist in systems engineering • Framework for linking multiple viewpoints • Framework for research • Link the design philosophy to NIST workin exchange form validation, requirements engineering, supply chain logistics simulation

4. What is special about V&V ? (1) • IBM Watson • New techniques – exceed human capability in knowledge-intensive tasks • “Machine understanding is not human understanding.” • “Knowledge is not the destination.”

5. What is special about V&V ? (2) • Validation & Verification • Knowledge is the destination. – knowledge, or at least credible rationale. • Requirement: • Minimally: be able to explain how the design space was characterized and demonstrate that requirements are being met. • Ideally: Provide deductive arguments where appropriate • Show how certain alternatives are indeed incompatible • Reference principles of operation, functions

6. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work

7. Basis for SE decision making • SE decision making macro level: • Trade studies, simulations, risk assessment, etc. • SE decision making micro level: • A web (conceptual schema) of information • Uncertain • Conflicting • Isolated • Uncertainty is quantified • Conflicts resolved • Inter-relation revealed

8. Strategy for the micro-level information • Characterize elements of rationale for SE decision making. • Each research project touches on only a few of these elements • No single overarching system design intended

9. 9 Elements of Rationale (1) • Measurement Conditions • Confidence in the process or environment under which it was measured • “Capacitance was measured using the AC impedance technique.” • Logical Consistency • Confidence due to consistency with theory. Type consistent. • “P = .05, as we’d expect from the law on conservation of energy.” • Associativity Across Views • Individuals: knowledge that two references made from different viewpoints refer to the same thing • “The region P on the CAD model corresponds to these elements in the FEA mesh.” (Individuals) • Concepts: knowledge that two conceptualizations can be used for the same purpose. • “What the supplier is calling ‘rated maximum pressure’ is what we call ‘rated pressure.” (Concepts)

10. 9 Elements of Rationale (2) • Change process • Knowledge of precursors and the history of properties that distinguish them. • “The value of P that we calculated for this design is close to what we found in earlier models.” • Authority • The power that information has due to an approval that is granted or an estimate of its maturity • “Supplier-provided data also suggest P=.05 is obtainable.” • Origin in Requirements* • Belief that a requirement is sensitive to it • “Our ability to achieve requirement x diminishes as P exceeds 0.07.”

11. 9 Elements of Rationale (3) • Origin in organization infrastructure • Belief because you obtained it in ways consistent with the organization’s best practices. • “P was obtained from the aero model in the preliminary design library.” • Consistency with other belief • Belief due to consistency with prevailing contingent facts • “P=0.5 is reasonable in products using component y.” • V&V Process • Belief that the system in place to manage the other 8 elements is sound and comprehensive. • “The value of P is confirmed through simulation that is routinely performed in validation of this product line.”

12. 9 Elements : Observations • Coupling and overlap • Authority / Origin in Organizational Infrastructure • Associativity across views / measurement conditions • etc. • Though these are found in models, they can be expressed from a more comprehensive viewpoint where • Contradictions can be exposed • Cohesion across views can be noted • Trace to requirements is more evident • (These are all parts of V&V)

13. MBSE Concepts / Logical View

14. Sentence detail / rationale

15. Example Usage Patterns • V & V • Origin in requirements • Automated generation of test cases • Requirements Engineering • Origin in other belief, emphasis on tracking contingent facts and engineering change • Refinement

16. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work

17. Exchange Form Validation : Two Methods • Axiomatic: • How: Map the exchanged content to sentences • Identify errors: ex falsoquodlibet with a reasoner • Advantage: Ontology explains intent • Disadvantage: Proofs hard to interpret • Metamodel: • How: Map the exchanged content to objects • Identify errors: Direct structural, with OCL, etc. • Advantage: Constraints relate to exchange form • Disadvantage: Constraints look like code

18. Example use of metamodel View / Viewpoint: Can be both consistent with a form (a view), and the form by which otherconceptualization are stated (a viewpoint.)

19. Example from the UML Metamodel

20. Example Specification Constraints

21. In MBSE, metamodels play a key role • Metamodel = (1) a specification of the form a model can take. (well-formedness conditions) (2) a formalization of the viewpoint that models will express

22. Metamodels also play a key role in model exchange • Metamodel = (1) a specification of the form a model can take. (well-formedness conditions) Definition of structure  serialization (2) a formalization of the viewpoint that models will express Illuminate what program structures the elementsof exchange content map to/from.

23. Communication with Exchange Standards

24. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work • Model Interchange Working Group • Supply Chain Logistics Simulation • Collaborative Requirements Engineering

25. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work • Model Interchange Working Group • Supply Chain Logistics Simulation • Collaborative Requirements Engineering

26. OMG Model Interchange Working Group • Goal: Improve the ability of OMG MOF-based tools (UML, SysML) to exchange information • XMI Serialization – common to MOF-based tools. • Process • Group: Produce Test Case diagram and reference file • Tool developers: create diagram in their tool, serialize as XMI • Use NIST tool to identify errors (in files and metamodels) • Correct tools and specifications

27. NIST UML / SysML Validator Enter below a file to upload:

28. NIST UML / SysML Validator

29. NIST UML / SysML Validator

30. NIST UML / SysML Validator

31. MIWG Results • Stakeholders witness significant improvement in interoperability Elaasar & Labiche, 2012

32. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work • Model Interchange Working Group • Supply Chain Logistics Simulation • Collaborative Requirements Engineering

33. Supply Chain Logistics Simulation • Goal: Demonstrate integrated use of models toward enterprise goals • Design: Map models, guided by metamodels into sentences that guide compilation of a discrete event simulation. • Models • UML of ordering / logistics objects • QVT-r mapping of messages to orders • BPMN “stereotyped” + OCL of business decisions • Discrete Event Simulation

34. Round Trip Engineering of Supply Chain Logistics

35. Logistics Processes (1)

36. Logistics Processes (2)

37. Business Rule

38. Simulation Results

39. Outline • Introduction / Scoping • Requirements for MBSE • Exchange Form Validation • NIST Work • Model Interchange Working Group • Supply Chain Logistics Simulation • Collaborative Requirements Engineering

40. Collaborative Requirements Engineering • Goal: Demonstrate Engineering from Product Data Sheets • Design: Map product data sheets in to sentences about requirements. Use these to guide engineering simulation and reasoning about alternative designs

41. Conclusions • Continuing roles for deductive reasoning in the automation of SE processes • The nature of V&V, requirements engineering, the way we think when we engineer, require it. • Preparing and interpreting macro-level SE decision processes is aided by the integration of multi-viewpoint, micro-level information. • Metamodels facilitate this integration.

42. References • Welty, C; Inside the mind of Watson, 2nd ESWC Summer School, Kalamaki, 2012, http://videolectures.net/eswc2012_welty_watson • Denno, P; Thurman, T, Mettenburg, J; Hardy, D; On enabling a model-based systems engineering discipline – 18th INCOSE International Symposium (2008) • Denno, P; Harrison, T; Using Legacy Modeling Artifacts in Supply Chain Logistics Simulation (in draft, 2013) • ISO 15288 (2008) – Systems and software engineering – System life cycle processes. (2008)