Formalized Model Development & Test Generation: Key Role of Abstraction

1. Formalized Model Development & Test Generation: Key Role of Abstraction Bernard P. Zeigler Arizona Center for Integrative Modeling and Simulation (ACIMS) www.acims.arizona.edu and Joint Interoperability Test Command (JITC) Fort Huachuca, AZ Foundations 04, ASU, Tempe, Oct.13-15

2. Outline • Systems-based framework for M&S • Bernard P. Zeigler and Hessam S. Sarjoughian, “Implications of M&S Foundations for the V&V of Large Scale Complex Simulation Models” Proc. Foundations 02, Laurel, MD, November, 2002, available from www.scs.org • Application to Systems Development • Interoperability Standards Conformance Testing • Role of Abstraction • Nature of abstraction, scalability • Abstractions employed in JITC application

3. Simulation-based V&V as an afterthought Government simulation tests of the system development of the system Contractor

4. Formalized Model System Development Process Reference Model Software- Intensive System Implement- ations Formalized Behavior Specification Simulation based testing Informal Behavior Requirements at one or more levels of System Specification Automated test suite design

5. Formalized Model System Development Process • Informal Behavior Requirements at one or more levels of System Specification: Systems theoryoffers well-characterized levels at which requirements for system behavior can be stated informally. • Formalized Behavior Specification: The behavior requirements are translated into a formalized model using framework for M&S methodology.. • Reference Model: serves as a reference model for any implementation of the standard. This model can be analyzed and simulated with a simulation protocol to study logical and performance attributes Using model continuity, it can be executed in real-time protocol and provides a proof-of-concept prototype for an operational system. • Automated test suite design: Branching in the lower path from the formalized specification, we develop a test suite consisting of test models within a test frame that can interact with a System Under Test (SUT) to test its behavior relative to the proposed standard. • Simulation based testing: The test suite is implemented in a distributed simulation infrastructure and executed against the SUT. The test suite provides explicit pass/fail/unresolved results with leads as to components/rules that might be sources of failure. • Optimization and Fielded Execution: The reference model provides a basis for correct implementation of the standard in a wide variety of technologies. The test suite provides a basis for testing such implementations as SUTs in a suitably extended test infrastructure. • Iterative nature of development: The process is iterative allowing return to modify the master formalized model and its informal precursor requirements. Model formalization minimizes the artifacts that have to be modified as the process proceeds.

6. How is simulation software different from other software? • It represents the behavior of dynamic systems whose state changes intrinsically depend on time • Properly controlling the flow of time is critical • Simulation software may combine: • continuous (time-driven) and discrete (event-driven) processes • actual operating hardware and software representations • wall clock and {faster/slower} than real time advance

7. independent specification of the dynamic system behaves over time DEVS A Formal Framework for Modeling and Simulation should provide methods needed to compare the behavior of the system implementation and the independent specification Experimental Frames

8. DEVS Background • DEVS = Discrete Event System Specification • Provides formal M&S framework: specification,simulation • Derived from Mathematical dynamical system theory • Supports hierarchical, modular composition • Object oriented implementation • Supports discrete and continuous paradigms • Exploits efficient parallel and distributed simulation techniques • Theory of Modeling and Simulation, 2nd Edition, Academic Press, 2000, Bernard P. Zeigler, Herbert Praehofer, Tag Gon Kim

9. DEVS Hierarchical Modular Model Framework Atomic: lowest level model, contains structural dynamics -- model level modularity Coupled: composed of one or more atomic and/or coupled models hierarchical construction + coupling

10. Formalized DEVS-based System Development Process Reference Master DEVS Model DEVS Model Continuity: System Implement- ations HLA Infrastructure Simulation based testing Captured As DEVS Temporal Rules Standards Document Mil-Std 6016C (JSLIRS) Automated test suite design: DEVS Federates

11. Interoperabilty Standards Test Generaton • Express the Mil-Std 6016C (JSLIRS) as a collection of state variables and transition rules • Rules are written as conditions, actions, and exceptions • If D is true now then do action E at time t, unless F occurs in the interim • State variables and rules characterize the dynamic behavior of the system • Basis for semi-automated reasoning tools to generate test cases that target specific system behavior

12. Transactions consist of sets of Rules Transaction Level - example P.1.2 = Drop Track Transmit 1 Preparation 2 Processing 3 Modify C2 Record for TN Transmit Msg Rule Processing Stimuli Constraints (Exception) Rules Validity checking Track Display Time outs Operator decisions Periodic Msg Other ConsequentProcessing Jumps (stimuli) to other Transactions of specification Stop Stop, Do Nothing, Alerts, Or jump to other Transaction

13. Abstractions: Needed For Scalability Base Specification Analyze useful information? (If you could) Preservation mapping inverse mapping Answers Abstraction Analyze • Different abstractions serve to address different questions • They may be sequenced combined, and crosschecked • Abstractions may be made more concrete

14. A C C D Rule 2 Rule 1 B Rules Dependency Abstraction Rule 2 depends on (can be affected by the execution of ) Rule 1 if there is a non-empty intersection between the action variables of Rule 1 and the condition variables of Rule 2

15. Dependency Abstraction and Concretization Manual Concretization Executable Rules Rule Engine Simulation RulesExpressed In textual form (XML) Feasibility Verification test plan candidate test paths Textual processing Dependency Graph Analysis Rule Abstractions

16. Example Dependency Graph start rule rule text ordinary rule output rule rule level dependencies Rule set: Operator action to initialize the terminal or change data

17. Example: Path Analysis • Rule test sequence: • Initialize Terminal • Check input data validity • Transmit J2.2 PPLI message shortest path (no cycles allowed within) rule under test test sequence for rule under test = feasible path from a start to an output that traverses the rule under test feasible = can be forced on the system under test by starting in a state (variablevalue assignments) that allows the designatied sequence of rule activiations to proceed feasibility has to be checked by rule engine

18. Abstraction via Modulo Operation on Variable Names G Modulo Message types Message Flow Analysis Message types G = Rules, Variables Dependency Graph Record types G Modulo Record types Data Store Dependency Analysis + invocations G Modulo Direct calls Invocation dependency Analysis

19. Abstraction via Rule Set Partitioning Hierarchical Modular DEVS G/group1 G = Rules, Variables Dependency Graph G/group2 G/group3

20. Employing Module Abstraction to Discover Test Paths • Module abstraction provides a “bird’s eye” view of the overall interdependence of transactions. • For example, we might seek a path that tests a desired sequence of transactions, and therefore, traverses a corresponding sequence of transaction modules • Such abstraction can eliminate large numbers of possible paths from consideration since paths not present at a higher level cannot appear at a lower level. • Moreover discovery of potential paths can be broken into two stages: • discover a path through modules • verify that internal connecting paths within the modules along this path exist • Potential paths need to be tested for feasibility using rule engine • For large rule sets, this multi-abstraction/multi-stage approach can significantly reduce the search necessary for path discovery.

21. Summary • A Formalized Model System Development Process provides the rigorous basis for specification of behavior. Such specification drives integrated and concurrent development of effective processes to test system implementations. • Test generation needs to exploit both micro and macro level analysis. Requirement might be for detailed local function testing or for overall sequence of behaviors • Abstraction is the key to the scalability needed to develop and test large complex systems • Multi-abstraction/multi-stage path discovery is a scalable methodology • We will apply these methods and tools to generate actual test cases, and conduct reviews by SME’s to guide further development • This formalized model, semi-automated test case development approach allows JITC to efficiently extract test cases from a standard, convert them to test procedures and apply these directly to a system for Standards Conformance and other testing efforts.