Overview of Cyber Experimentation & Test Ranges

1. Overview of Cyber Experimentation & Test Ranges ICOTE September 252012 William C. Liu Section Lead Cyber Operations & Networking Group BAE Systems – Technology Solutions Arlington VA 22203 USA Dr. Kevin M. McNeill Engineering Fellow & Technical Director Cyber Operations & Networking Group BAE Systems – Technology Solutions Arlington VA 22203 USA Log# ES-AVA-040912-0073 Approved for public release 0162; No Export Controlled Data 1

2. Problem Domain - overview • Software is in everything! • Functionality defined by software (cyber) but tight coupling to physical (kinetic) world • Design flaws, unintended bugs, vulnerabilities or attack have physical effects • Large-scale, complex cyber-physical systems and systems-of-systems across a wide variety of application domains • Such systems deployed for mission-critical operations in many domains, e.g.: • Defense • Critical Infrastructure • Health-care • Finance Source: IEEE Computer Magazine 2009 Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 2

3. Problem Domain – Broad Challenge • General purpose methods and tools supporting development of the software (cyber) element of complex cyber-physical systems fail to - • provide assurance for mission-critical functions • provide assurance the system will satisfy safety and reliability requirements • provide assurance the system will support scalability and adaptability demands • provide support for the analysis of vulnerabilities to offensive cyber operations • a), b) & c) belong to a broad software engineering challenge for test and validation that is beyond the scope of this presentation! d) introduces additional challenges for test and validation that must be addressed by Cyber Experimentation & Test Ranges Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 3

4. Problem Domain – Cyber Testing & Validation • General technical approach to testing, validation & verification of these cyber-physical systems are not adequate • Especially with respect to analysis related to cyber operations • Common methods and techniques do not support – • Rigorous scientific/engineering methodologies • Rapid configuration and adaptation of test environments • Flexibility of test environments to accommodate new technologies • Automation of test processes • The result is that cyber test ranges are often expensive, unreliable and quickly become obsolete This is the testing challenge that is the focus of this presentation! Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 3

5. Why is it a problem? • New generations of distributed ultra-large scale, systems of systems… • Global Enterprise Networks; Command & Control Systems; Smart-Grid; Next Generation Air Traffic Control; Cyberspace operations… • … are complex systems… • Often built from rapidly evolving open-source, Internet-based services • Driven by user demands for rapid evolution (or changing mission requirements) • Have complex configuration options • Require a capability to respond to unexpected situations/environmental factors • Contain unknown software vulnerabilities and are often accessible from around the globe via Internet • Rigorous testing, validation & verification of these systems based on scientific & engineering principles is not supported by legacy tools • Cost (time) to develop, predictability of behavior in unexpected conditions, safety, reliability, identification of vulnerabilities, verification of requirements Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 4

6. Focus on Cyber Experimentation & Test Ranges The growth of cyber intrusions presents a key challenge to national and international scale enterprises in their deployment and operation of large-scale cyber-physical systems • Cyber experimentation & test ranges are used to understand vulnerabilities of computing and networking infrastructure • Critical Challenges • Building and operating these ranges is complex, costly, time-consuming and often does not provide sufficient fidelity to provide any real value • Cyber testing challenges are made even more complex by the nature of rapidly of services provided by centralized service provides (e.g., ISP’s, ASP’s) that support many business operations Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 5

7. Cyber Experimentation & Test Challenges (1) Live testing on the operational or development networks of an Enterprise is not recommended due to risks to business operations • Therefore, cyber testing must be conducted on highly isolated environments • To build a successful, useful & maintainable cyber experimentation and test range it is necessary to address key challenges - • Scope – Identify the scope of the experimentation & test to size and equip the range infrastructure appropriately (do you need a few servers & VM’s or a data center?) • Central to estimating personnel and on-going costs (e.g., technology refresh, licensing costs, training and operations) • Replication – Enterprises use services provided by centralized providers, potentially distributed across multiple data centers or provided by Cloud services • It is not economically feasible to recreate these in a range with a high degree of fidelity Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 6

8. Cyber Experimentation & Test Challenges (2) • key challenges (cont’d) - • Proprietary – Environment under test may use proprietary software that is not available for internal analysis or testing • No private instances of the commercial services are allowed for analysis, experimentation and test • Licensing – Implementing a large-scale cyber range that to replicate an enterprise for network vulnerability assessment may require a large number of virtual machines running commercial software • Vendors may require paying for a license for each instance active on the range • Adaptability – Enterprises leverage new services that come online and become important operationally very quickly and often change rapidly • Cyber ranges must facilitate rapid integration or replication of new services to provide a realistic test environment Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 7

9. Cyber Experimentation & Test Challenges (3) • key challenges (cont’d) - • Contamination – Resetting the range to a pristine state after testing must ensure that no latent malware remains or can propagate off of the range • Requires policies, procedures and technologies to verify the state of the range before, during and after the test • Sensors/Instrumentation and visualization of the range are critical • This is a very hard problem • Fidelity – Replicating the hardware, software and network environment in a way that provides sufficient realism is necessary to ensure that the range produces test results that are meaningful • Difficult challenges relate to scale, replication of user traffic, emulation of user behaviors, or use of non-standard hardware • Operations & Personnel – Need to have a business model for the range “enterprise” • Who are the “customers” for the range; • What are roles and responsibilities for the range personnel; • What is the cost/revenue model for ongoing maintenance and support; Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 7

10. Technologies to overcome these challenges (1) Model-based Automation Frameworks • To avoid complete duplication of an enterprise environment for testing cyber range operators must use a combination of live/virtual/constructive testing with significant emulation/simulation or analysis • Solving this problem for a specific class of systems is feasible and well supported by the using a model-based, adaptive testing framework • Such a framework should support multiple levels of abstraction to accommodate different testing requirements and phases • It should be model-based (e.g., using domain-specific modeling languages) to support rapid reconfiguration and update • It should integrate with custom emulation tools for non-standard systems • It should be implementing to use virtualization to a great extent in order to facilitate scalability and cost management Domain-specific modeling languages (DSMLs) allow definition of models for individual aspects of cyber testing and support automation and rapid adaptation Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 8

11. Technologies to overcome these challenges (2) Hierarchical Levels of Abstraction to Address Complexity • Isomorphic Testing – The cyber test range can be configured with analogous software that can be shown to have features and behaviors that map onto those of the application of interest • Behavioral Models - This level of abstraction may include some form of transaction replay by a model of the application of interest • Simulation - For cyber testing this provides relevance in assessing scalability and interaction of many actors across an large scale networking environment or assessing Enterprise impact • Emulation – Provide a higher level of fidelity or adaptability and works well when internals are a “black-box” and the interaction with the external network is important Model-based Automation Frameworks, built with virtualization technologies and supporting various levels of abstraction significantly reduce the time, cost and complexity of conducting cyber experimentation and test exercises while supporting scientific and engineering rigor Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 9

12. Automation of the test processes (1) Most cyber experimentation and testing is a sequence of highly manual processes that are very time consuming, costly and prone to error Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 10

13. Automation of the test processes (2) • Using Domain Specific Modeling Languages (DSML) allowsmodel-based test tools to be tailored to specific user needs • Model representation supports re-use of test recipes, enforcement of Information Assurance policies and the application of scientific rigor • Transitions between steps become model transformations that can be automated to increase R&D testing throughput • Test evolves from abstract concept to physical instantiation Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 11

14. Cyber Experimentation & Test Process Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 12

15. Model-base Cyber Experimentation & Test Capability • Model-based Range Operations concept showing technical aspect of the cyber range operations • Not shown are business processes associated with range operations or tools associated with inventory management, resource control and technology refresh Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 13

16. Summary • The model-based automation framework provides a flexible infrastructure for the development and testing of large-scale complex software systems • The framework itself is an adaptive, model-drive system that is very flexible • New DSML’s can be rapidly created to allow the framework to support domain specific testing requirements • We have demonstrated its use for cyber experimentation & test research • This framework extends the existing paradigm of Live/Virtual/Constructive simulation by applying the model-driven approach to the entire testing process • Test planning, test deployment, test execution, range operations & test-recipes • The process is realized as a set of model transformations and provides enhanced support for scientific rigor and application of domain specific constrains (e.g., IA) Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 14

17. Academic Research • Vanderbilt University, Institute for Software Integrated Systems (ISIS) is a world leader in the research and development of Model-based Engineering and Model-Integrated Computing Tools • University of Utah EMULAB is a leader in the development of large-scale virtualization ranges for network and software research, especially for tools that manage the deployment of experiments onto the range Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073

18. QUESTIONS? Approved for public release 0162; No Export Controlled Data • Log# ES-AVA-040912-0073 15