M&S–Based System Development and Testing In a Net-Centric Environment

1. M&S–Based System Development and Testing In a Net-Centric Environment Bernard P. Zeigler, Ph.D., Arizona Center for Integrative Modeling and Simulation and Joint Interoperability Test CommandFort Huachuca, AZ 85613-7051zeigler@ece.arizona.edu

2. Motivation from a Testing Perspective • Traditional interoperability concepts and test practices are facing severe challenges from several sources: • a) complexity within each new system, as well as composition into families of systems and systems of systems, • b) extensive use of modeling and simulation in simulation-based acquisition, and • c) the move to service oriented architecture (SOA) to support composable services over the Global Information Grid (GIG). • Need a methodology and processto integrate development and testing in a net-centric environment • combines with and goes beyond current software development paradigms • rests upon and exploits dynamic systems theory, a modeling and simulation (M&S) framework, and model-continuity development concepts • Relationship to other software development processes will be open for discussion at the end

3. Part 1 Modeling and Simulation Background Briefly review the M&S framework and theory • provide background for integrated system development and testing process • overview the Discrete Event Systems Specification (DEVS) formalism which provides the computational support for the M&S framework and theory • DEVS can serve as the basic medium for formalization of standards, analysis of the resulting models, automation of the test cases, and execution of the test drivers

4. Part 2. M&S-Based Development and Testing: DEVS Methodology • Illustrate M&S-Based testing methodology with an application drawn from a current effort to provide automated test generation for an important tactical command and control standard, MIL-STD 6016c • Goal: develop a “formalized” version of the MIL-STD 6016C rule sets with the objective • to complete an unambiguous description of the specification, • enable more automated test development • Result: Automated Test Case Generator (ATC-Gen) is a methodology and tool-set based on the DEVS formalism. • Success: ATC-Gen will be one of the primary test vehicles for the first assessment of an integrated air picture system based on MIL-STD 6016c to occur this fall

5. Framework for M&S Systems theory-based framework • provides an ontology for M&S that recognizes the essential dynamic character of simulation models • distinguishes the elements in the M&S enterprise (such as models, simulators, and experimental frames) and the relationships (such as validity and correctness) that connect such elements in meaningful ways related to the objectives of simulation exercises, • provides a rigorous mathematical theory that supports manipulations of the elements in their real-world incarnations in order to achieve the desired relationships

6. M&S Entities and Relations Device for executing model Real World Simulator Data: Input/output relation pairs modeling relation simulation relation Each entity is represented as a dynamic system Each relation is represented by a homomorphism or other equivalence Model structure for generating behavior claimed to represent real world

7. d q(t) / dt = x(t) M&S Framework: Continuous case Real World Simulator modeling relation simulation relation • Numerical Integration: • Accuracy • Error effects Model • Validity: • Accuracy of • -retro-diction • -prediction

8. output x x 0 1 third Duration second Duration X 0 First Duration first second passive third t1 t0 t2 active s’ Time advance s Make a transition M&S Framework: Discrete Event case Real World Simulator simulation relation modeling relation • Validity: • Accuracy of • -retro-diction • -prediction • Concurrent Processing • Correctness • Randomness Model

9. DEVS within the Framework for M&S • DEVS (Discrete Event Systems Specification) formalism • is a constructive framework for M&S • based on mathematical systems theory • DEVS enables applications resting on rigorous theory • modeling discrete/continuous heterogeneously composed networked systems – universality of representation • hierarchical, modular model construction –closure under coupling • verifiably correct parallel and distributed simulation of ultra-large scale networks • model-continuity from simulation to execution – based on abstract simulator • automated test generation for net-centric services – based on behavior representation from systems theory

10. DEVS Hierarchical Modular Composition – Key to Constructing/Orchestrating Complex Systems Atomic: lowest level model, contains structural dynamics -- model level modularity Coupled: composed of one or more atomic and/or coupled models hierarchical construction + coupling

11. Types of Models and their DEVS Representations Coupled Models Atomic Models Partial Differential Equations Ordinary Differential Equations Processing/ Queuing/ Coordinating Networks Collaborations Physical Space Phase Based Models Pulse Based Models (varGen, Sum) Digraph Models 1,2 Dim Cell Space Discrete Time/ Automata Quantum Based Models (DEVS Integrator, instantaneous Functions Cellular Automata can be components in a coupled model 2 Dim State Space 1 Dim State Space Self Organized Criticality Models Multi Agent Systems

12. DEVS Theory: Nutaro's optimistic simulation algorithm and its correctness proof • Nutaro developed a time warp variant that correctly simulates all discrete event models • Previous variants of the time warp algorithm have been based on the logical process world view. • These time warp derivatives have been widely used for simulating discrete event models on parallel computers, • But, they can not correctly simulate DEVS coupled models with zero-time interactions • employs a 2-D distributed clock • A correctness proof for the algorithm was constructed • The correctness proof demonstrates that the parallel algorithm simulates exactly the same system as the canonical DEVS abstract simulator. • Consequently, parallel and sequential simulation of the same DEVS model will always produce the same results. • This is a strong proof of correctness that no logical-processor-based proof has been able to rival – depends strongly on model/simulator separation

13. Robot Robot Robot Real Real Real Model Model Model Robot Robot Robot Virtual Virtual Real Real Virtual HIL Virtual Virtual Sensor Actuator Sensor Actuator Sensor Actuator Sensor Actuator Virtual Environment Mixed Environment Real Environment DEVS Distributed RT Executor DEVS Distributed Simulator DEVS-Based Model Continuity • Allows component models of a distributed real-time system to be tested incrementally then deployed to a distributed environment for execution • Based on replacing DEVS simulator with DEVS Real Time executor • Model continuity is retained between models of different design stages • reduces the occurrence of design discrepancies along the development process • increases the confidence that the final system realizes the specification

14. Model Continuity Mixed Virtual/Real Environment – Any number of virtual robots can interact with a few real robots

15. Summary DEVS is an overarching approach • supports all levels of system interoperability • using a Systems Theoretic worldview • component-based characterization of dynamical systems in discrete and continuous forms • methods for modular, hierarchical model composition targeting reusability and composability

16. Part 2. M&S-Based Development and Testing: DEVS Methodology • DoD acquisition policy requires testing throughout the systems development process to ensure interoperability and conformance to standards. • The Joint Interoperability Test Command (JITC) is striving to improve traditional standards conformance and interoperability testing methodologies to • keep pace with the behavioral complexity of new systems that are increasingly reliant on advanced information technology • DoD mandated use of M&S throughout the development life cycle, requires testing methods that are themselves M&S-based • In 2003, the JITC started development of a “formalized” version of the MIL-STD 6016C rule sets with the objective • to complete an unambiguous description of the specification, and • enable more automated test development. • illustrate the methodology with an application drawn from a current effort to provide automated test generation for an important tactical command and control standard, MIL-STD 6016c • Automated Test Case Generator (ATCGen), which is a methodology and tool-set based on the DEVS formalism. • ATC-Gen will be one of the primary test vehicles for the first assessment of an integrated air picture system based on MIL-STD 6016c to occur this fall. • working towards a proposal for how the DEVS-based approach enables • 1) simulation-based model continuity in developing and testing specifications that stem from the DoD Architectural Framework (DoDAF), and • 2) a standard and an infrastructure for developing and testing web-based services for DoD’s emerging Net-Centric Enterprise Services on the Global Information Grid.

17. Outline • JITC responsibility for Link-16 and successors • Link-16: Challenges to Implementation and Testing • M&S–based Approach to Automated Test Case Generation • Application to Link-16 in the IABM SIAP Context

18. JITC is the Responsible Test Organization for TDL Standards • JITC is responsible for ensuring systems that implement Tactical Data Link (TDL) (Link 11/11B/16 and Variable Message Format (VMF)) are interoperable and in compliance with the applicable joint standards. • This is accomplished by conducting the following types of tests: • Joint / NATO /Combined Interoperability • Performance Assessment in Operational Environments • Standards Validation • Standards Conformance • JITC employs a variety of tools that provide its analysts the ability to evaluate TDL system performance in both the lab and live environments. source: http://jitc.fhu.disa.mil

19. Link-16: Challenges to Implementation and Testing Joint Single Link Implementation Requirements Specification JSLIRS is an evolving standard (MIL-STD-6016c) for tactical data link information exchange and networked command/control of radar systems • Presents significant challenges to automated conformance testing: • The specification document states requirements in natural language • open to ambiguous interpretations • The document is voluminous • many interdependent chapters and appendixes • labor intensive and prone to error • potentially incomplete and inconsistent. • Problem: how to ensure that a certification test procedure • is traceable back to specification • completely covers the requirements • can be consistently replicated across the numerous contexts • military service, inter-national, and commercial companies

20. SIAP/IABM —Successor to Link-16 • SIAP (Single Integrated Air Picture) Goal: Improve the adequacy and fidelity of information to form a shared understanding of tactical situation • Integrated Architecture Behavior Model (IABM) requires that all sensors utilize a standard reference frame for conveying information about the location of targets. • Developed by the Joint SIAP System Engineering Organization (JSSEO), Arlington, Va., a sub-office of the Assistant Secretary of the Army for Acquisition, Logistics and Technology. • Development costs $160 million over five years; the individual services will spend an estimated $600 million • First major test – Config05 – next week -- ATC-Gen will be the basis for testing link-16 and extended IABM requirements http://www.navyleague.org/sea_power/mar_04_18.php

21. Automated Test Case Generator (ATC-Gen) • IABM is an extension of Link-16 developed in HLA environment and requires HLA simulation-based testing • JITC has taken the initiative to integrate modeling and simulation into the automation of the testing process • Funded the development of Automated Test Case Generator (ATC-Gen) led by ACIMS using DEVS (Discrete Event System Specification) technology. • In R&D of two years, proved the feasibility and the general direction • The requirements have evolved to a practical implementation level, with help from conventional testing personnel.

22. Transaction Level - example P.1.2 = Drop Track Transmit 1 Preparation 2 Processing 3 Modify C2 Record for TN Transmit Msg Rule Processing Constraints (Exception) Rules Validity checking Track Display Time outs Operator decisions Periodic Msg Other ConsequentProcessing Jumps (stimuli) to other Transactions of specification Stop Stop, Do Nothing, Alerts, Or jump to other Transaction Output from Input to system system DEVS t t t t 1 2 3 4 Discrete Event Nature of Link-16 Specification

23. System Under Test (SUT) Test Driver DEVS Simulator HLA HLA Network Automated Test Case Generation: Goals and Approach • Goals: • Increase the productivity and effectiveness of the conformance testing • Apply DEVS systems theory, modeling and simulation concepts, and current software technology to (semi-)automate portions of conformance testing Objective: Automate Testing Capture Specification as DEVS Analyze DEVS I/O Behavior Synthesize Test Models Test Driver executes models to induce testable behavior in SUT Formalized approach for converting standards documents into test models to run directly against a system, automating the process to the extent possible Interact with SUT over Middleware

24. Benefits of Formalization and Automation • Provides traceability to original specification • Reduces ambiguity from textual specification • Facilitates integrating Modeling & Simulation into the testing process • Enables testing of complex: • Standards • Systems • Functions • Families of systems

25. ATC Gen Overall Process • MIL-STD-6016C is a description of a DEVS that mandates the outcome of tests and sequences of tests • ATC Gen analysts convert if-then rules from the MIL-STD into XML, defining state variables and vectors • XML if-then rules are used to generate test sequences • Test models are derived from test sequences • Test Driver runs test models against SUT in distributed simulation

26. ATC-Gen Process Lines XML XML Rule Set 6016C Rule capture and Dependency Analysis Rule Set Analyze Rules Define State Variables XML Rule capture Rules Rule Reference Model Executable Compiler paths Rule Formalization Compile executable Reference model Test Driver Verification of Test Sequences Test Generation Test case models Test Stimulator Executable Test sequences Simulations of test cases Reference model Tests Logging Test Execution Test Results Test Procedures • The Dependency Analyzer (DA): • determines the relationship between rules by identifying shared state variables • Generates test sequences and test models • Analyst: • Interpret the text to extract state variables and rules, where rules are written in the form: • If P is true now • then do action A later • unless Q occurs in the interim • Test Engineer: • Develop sets of test sequences • Convert test sequences to executable simulation models • Convert test sequences to executable simulation models • Test Driver: • Interacts with and connects to SUT via HLA or Simple J interfaces • Performs conformance testing of SUTs

27. MIL-STD-6016C Micro-Structure Message Phase Message Transmission Message Preparation Message Reception • Stimuli • Constraints • Processing Appendix Structure Roles Initiator Responder

28. Interpretation and Translation

29. Interpretation and Translation

30. Detailed XML Example: Traceability, Quality Assurance 4.11.13.12Execution of the Correlation. The following rules apply to the disposition of the Dropped TN and the retention of datafrom the Dropped TN upon origination or receipt of a J7.0 Track Management message, ACT = 0 (Drop Track Report), for the Dropped TN. The correlation shall be deemed to have failed if no J7.0 Track Management message, ACT = 0 (Drop Track Report), is received for the dropped TN after a period of 60 seconds from the transmission of the correlation request and all associated processing for the correlation shall be cancelled. a. Disposition of theDropped Track Number: (2) If own unit has R2 for the Dropped TN, a J7.0 Track Management message, ACT = 0 (Drop Track Report), shall be transmitted for the Dropped TN. If the Dropped TN is reported by another IU after transmission of the J7.0 Track Management message, own unit shall retain the dropped TN as a remote track and shall not reattempt to correlate the Retained TN and the Dropped TN for a period of 60 seconds after transmission of the J7.0 Track Management message. • XML Translation: <rule trans="4.11.13" stimulus="4.11.13.12" reference="4.11.13.12.a.2" ruleName="R2 Unit transmits J7.0"> <condition txt="Check for R2 own unit" expression="AutoCor==True and (CRair.TNcor.CORtest==3 and J32.TNref.CORtest==3) and CRair.R2held==1 AND J72.MsgTx==True"> </condition> <action txt="Prepare J7.0 Drop Air Track message" expression="J70.TNsrc=TNown; J70.TNref=TNdrop; J70.INDex=0; J70.INDcu=0; J70.ACTVair=0; J70.SZ=0; J70.PLAT=0; J70.ASchg=0; J70.ACTtrk=0; J70.ENV=0; MsgTx(J70)"> </action> <output txt="Transmit J7.0" outType="Message" outVal="J70"></output> </rule> <QA> <revisit name="DHF" date="10/16/04" status="Open">need to add timer for a period of 60 seconds in which correlation is not reattempted</revisit> </QA>

31. Quality Assurance (QA) • Provides a history of rule changes • Identifies possible Change Requests and other problematic portions of the standard • QA tags: • Info: Explanation / reasoning for rule interpretation • Revisit: Rule needs further analysis; set to Open or Closed • Resolution: Solution to question / issue

32. MIL-STD-6016C Macro-Structure • Transactions are atomic units • Functions are collections of Transactions • Appendices group related Functions • Java processing restructures rules into a Transaction Module Hierarchy AppendixNFunctional Area, e.g. Track Management FunctionP.N e.g. C2 Drop Track TransactionP.1.N C2 Drop Track Transmit Atomic Level Unit Step 1 – Stimuli, Preparation, Constraints Step 2 – Processing, Operator Decisions Step 3 – Update Database, Output

33. XML Repository – GIG/SOA Asset • Organized according to MIL-STD-6016C macro-structure hierarchy • RuleCombo folders store aggregations/abstractions of lower level rules • Value added: • Provides a basis for further analysis • Removes ambiguity • Annotates problems areas, improving the ability to find and fix issues • Provides organization for indexing states, rules, and variables • Supports test generation and executable rule construction

34. Dependency Analysis:Visualization of Rule Dependencies Clicking will reveal rule text Dependencies revealed by hovering

35. Test Sequence Generation Transaction Level Paths C.1ModuleN active  = infinity D.1ModuleN active  = infinity Dependency Between Modules Potential Test Sequence U.1ModuleN passive[out1]  = infinity Desired Sequence C.1 Receive PPLI D.1 Receive Track U.1 Correlate

36. Determining initial state and message field values required to drive SUT through sequence Analyst: • Determine the data needed to execute a test sequence • Set state variables and field values accordingly

37. Test Simulatlon Architecture Test Model System Under Test J-Series Message Acceptor J-Series Message Generator Checks for Proper Response Produces Stimulus

38. Test Model • holdSend(Jx1,data1,t1) • holdSend (Jx2,data2,t2) • holdSend (Jx3,data3,t3) • waitReceive(Jx4,data4) Jx1,data1 Jx2,data2 Jx3,data3 Jx4,data4 t1 t2 t3 t4 time receiveAndProcess(Jx1,data1) receiveAndProcess(Jx2,data2) receiveAndProcess(Jx3,data3) transmit(Jx4,data4) SUT Test Model Construction: Translate test sequences from the DA to derive a composed model for the Test Driver

39. Test Driver Composition and Deployment Test Model Sequence 1 Sequence 2 Sequence 3 Jx1,data1 Jx2,data2 Jx3,data3 Jx1,data1 Jx2,data2 Jx3,data3 Jx1,data1 Jx2,data2 Jx3,data3 Jx4,data4 Jx4,data4 Jx4,data4 Middleware SUT

40. IABM/Link 16 Correlation Testing • ATC Gen automated correlation/decorrelation tool provides in-depth compliance testing • Mathematical algorithm of the proximity of the tracks • Logic for prohibitions and constraints • Test for post-correlation (dropped and retained tracks) • Discriminates between types of failures encountered during correlation testing

41. ATC Gen Summary: What Can (not) be Automated?

42. Discussion Can the DEVS-based M&S approach enable • simulation-based model continuity in developing and testing specifications that stem from the DoD Architectural Framework (DoDAF), and • a standard and an infrastructure for developing and testing web-based services for DoD’s emerging Net-Centric Enterprise Services on the Global Information Grid.

43. SystemDevelopment & Demonstration ConceptRefinement TechnologyDevelopment Production & Deployment Operations & Support IndustrySystemLife-Cycle Concept Analysis Design Develop Test Deploy Maintain ProofofConcept DiagnosticEmulation SystemSimulation SystemsEmulation ModelBased LVC SimulationDemos LVCTraining ConceptsTesting PrototypeTesting DesignTesting SystemValidation SystemVerification UseabilityTesting DiagnosticTesting Systems Engineering Perspectives for M&S Applications (Robert Marcus) DoD System Life-Cycle

44. Interlocking Standards – How Do These Play Together? SystemDevelopment & Demonstration ConceptRefinement TechnologyDevelopment Production & Deployment Operations & Support IndustrySystemLife-Cycle Service Testing Service Composition Concept Analysis Design Develop Test Deploy Maintain Service Discovery Service Description ProofofConcept DiagnosticEmulation SystemSimulation SystemsEmulation ModelBased LVC SimulationDemos LVCTraining Packaging Experimental Frame Messaging Model ConceptsTesting PrototypeTesting DesignTesting SystemValidation SystemVerification UseabilityTesting DiagnosticTesting Communication Simulator Architecture Framework Standards DODAF Operational View Systems View Technical View Development Process Standards Hierarchy of System Specifications Service Oriented Architecture Standards Middleware Standards M&S Standards

45. Models and Methodologies –How Do These Play Together? The MDA defines an approach whereby you can separate the system functionality specification from its implementation on any specific technology platform. That way you can have an architecture that will be language, vendor and middleware neutral. MDA Model Driven Architecture Petri nets were invented by Carl Adam Petri to model concurrent systems and the network protocols used with these systems. Petri Nets IDEF Integrated Definition for Function Modeling UML Unified Modeling Language MDD Model Driven Development IDEFØ is a method designed to model the decisions, actions, and activities of an organization or system. IDEFØ was derived from a well-established graphical language, the Structured Analysis and Design Technique (SADT). The United States Air Force commissioned the developers of SADT to develop a function modeling method for analyzing and communicating the functional perspective of a system. Effective IDEFØ models help to organize the analysis of a system and to promote good communication between the analyst and the customer. IDEFØ is useful in establishing the scope of an analysis, especially for a functional analysis. As a communication tool, IDEFØ enhances domain expert involvement and consensus decision-making through simplified graphical devices. As an analysis tool, IDEFØ assists the modeler in identifying what functions are performed, what is needed to perform those functions, what the current system does right, and what the current system does wrong. Thus, IDEFØ models are often created as one of the first tasks of a system development effort. "model-driven" development methods, based on greater use of automation compared to traditional methods, have already demonstrated their potential for radical improvements in the quality of software This is a 4GL notation tailored to OO software development. It is primarily a graphical notation. The standard is managed by the OMG.integrates the concepts of Booch, OMT and OOSE by fusing them into a single, common and widely usable modelling language. UML aims to be a standard modelling language which can model concurrent and distributed system

46. Ideal Bifurcated Development Process Reference Master Model Simulation based testing in net-centric environment Formalization by mapping into DEVS representations Corresponding levels of System Specification Real-time implement-ation and execution Behavior Requirements at one or more levels of System Specification Semi-automated test suite design based on Experimental Frames

47. Bifurcated Development Process in the DODAF/GIG Context WEB SERVICES OV-8 information mapped to WSDL and vice versa. Any Web Service can become a component Activity of OV-8 and be constituted as a ‘functionality’ in DoDAF OV Activities classified into Activity Components Activity characterization and System scope refinement Integrated Functionalities transported to System views using: 1. Model Continuity 2. Web services 3. COTS components OV-3,8,9 DEVS Formalization OV-7,4 AV-1,2 OV 1,5,6,2 Simulation based testing using DEVS Test Federations Semi-automated test suite design based on NLspecified vignette scenarios Model Design & Repository

48. DEVS Model DEVS Simulator HLA DEVS Model Packaging:FOM DEVS Simulator Messaging:Interactions,Updates Communication: RTI SOA Service Discovery: UDDI Sevice Description: WSDL Packaging:XML Messaging:SOAP Communication: HTTP Transfering DEVS-based Testing Methodology to SOA

49. Refining OV-6a (IDEF) to OV-8 (DEVS) IDEF • Inherent problems with IDEF process (&,||,!), • Timing, a critical factor in real-time decision making • unaddressed by CPNs, IDEF processes, Automated Code generation of DEVS model thru OV-8 DEVS • LOGIC CODE IN OV-6a • (Rules of Engagement/Activation of further Activities done with boolean decision making) • IF Significant Movement of target • Then Monitor Target/Target Status • Project Target Movement • Target Vector = . . . .. ? • Else • Monitor for Movement

50. DEVS Test Player Service Under Test DEVS Test Federation GES Live Test Player DEVS Simulator Node SOAP-XML DEVS-based Web-Services Testing