Model-Validation in Model-Based Development

1. Model-Validation in Model-Based Development Kurt Woodham L-3 Communications Ajitha Rajan, Mats Heimdahl University of Minnesota OSMA SAS ’08 September 8-12

2. Problem: Model Validation • Model-Based Development (MBD) is here to stay • Use of MBD is accelerating • Estimate 50% of NASA development projects using some form of MBD • Many advantages: model-checking, code generation, desktop testing, closed-loop simulation • Enhances early detection of requirement, design, or implementation defects • “Executable Specifications” enable evaluation of behavior that might otherwise be relegated to Inspections and Testing • How do we know the models are “right”? • Manually develop black-box tests • When have we validated enough? • Measure test coverage on an implementation/model SAS_08_Model_Val_Exec_Heimdahl

3. Problem : Current Practice • Measure black-box test coverage over the model • Indirect measure • Defects of omission in model not exposed. • Executable artifact is necessary • Adequacy can only be determined late in the development process Incomplete Model Weak Black-Box Test set SAS_08_Model_Val_Exec_Heimdahl

4. Goals of Project • Define metrics for objective, implementation-independentmeasure of adequacy of a black-box test suite • Develop tools to measure validation adequacy based on the defined metrics • Provide capability for autogeneration of black-box test suites SAS_08_Model_Val_Exec_Heimdahl

5. Assertions Does it implement? Specification Does it implement? Model Implementation Model-Based Testing (MBT) to Verify Code Model Does it implement? Source Code Testing – What does it mean? Assertion Based Testing (ABT) to Validate Model In General Our contribution is in providing novel ABT capabilities SAS_08_Model_Val_Exec_Heimdahl

6. What are Assertions? Properties/ Formal Assertions Assertions Can also be over components, interfaces,... Defined over System in1 out1 ink outm SAS_08_Model_Val_Exec_Heimdahl

7. Validate Assess Model and Assertion Completeness 3 Model Contributions - ABT 2 Auto-generate Black-Box Tests Assertions Measure Adequacy 1 • We provide the following contributions in the Assertion-Based testing domain (indicated by in the above figure): • Objective, implementation-independentmeasure of adequacy of a black-box test suite • Auto-generation of black-box validation tests directly from assertions • Objective assessment of completeness of model as well as assertions SAS_08_Model_Val_Exec_Heimdahl

8. Relevance to NASA • MBD is here - estimate one-half of all NASA missions in development or on the books will use model-based subsystem development • Extensive use in avionics industry • How do we know the models are right? • Model validation problem • We provide the capability to • Objectively measure the “quality” of assertion-based black-box validation tests • Objectively assess the completeness of a model • Does the model address all assertions? • Objectively assess the adequacy of a set of assertions • Are there enough assertions to adequately describe the model? • Automatically generate truly assertion-based tests SAS_08_Model_Val_Exec_Heimdahl

9. Achievements to Date • Formal assertion notation identified • Most work with LTL • Extended to work with Live Sequence Charts (LSC) • Objective validation metrics defined • Requirements, Antecedent, Unique First Cause, and Unique Cause • Test case generation tool developed • Developed tool generating tests from LTL • Capable of generating tests to all metrics defined • Prototype tool working on LSC developed • Developed test-adequacy measurement tool for the defined validation metrics • Evaluation of metrics and tool • 12 papers and one PhD dissertation (Ajitha Rajan) Tool Demo in Tech. Session SAS_08_Model_Val_Exec_Heimdahl

10. Next Steps • Investigate alternative requirements notations to LTL • Complete empirical evaluation of the effectiveness in model validation • Flight Guidance System (FGS) evaluation complete • Display Manager (DM) evaluation in work • Coordinate evaluation on NASA IV&V project • Coordinate technology transfer SAS_08_Model_Val_Exec_Heimdahl