Exercises in Defending Cyberspace: The Capstone of Education, Training, and Awareness

1. Exercises in Defending Cyberspace: The Capstone of Education, Training, and Awareness Craig E. Kaucher LTC, U.S. Army Professor of Information Operations and Assurance Information Resources Management College National Defense University kaucherc@ndu.edu “My opinions: not necessarily the USG, DOD, or NDU!”

2. Agenda • Why Exercises ? • Developing Exercises • Sponsor • Objectives • Scope and Format • Participants • Scenario • Controllers • Models • Testing and Validation • Case Studies

3. Why Exercises ? • “To test civilian agencies’ security preparedness and contingency planning, DHS will use exercises to evaluate the impact of cyberattacks on governmentwide processes. Weaknesses discovered will be included in agency corrective action plans and submitted to the OMB. DHS also will explore such exercises as a way to test the coordination of public and private incident management, response and recovery capabilities.” (A/R 1-3) • “Corporations are encouraged to regularly review and exercise IT continuity plans and to consider diversity in IT service providers as a way of mitigating risk.” (A/R 1-4) Appendix, Actions and Recommendations (A/R) Summary, The National Strategy to Secure Cyberspace, February 2003.

4. “To rely on rustics and not prepare is the greatest of crimes; to be prepared beforehand for any contingency is the greatest of virtues.” Sun Tzu, on the need to wargame strategies, from “Sun Tzu and the Art of Business: Six Strategic Principles”, Mark McNeilly, Oxford University Press, 1996. Why else…(Obligatory Dead Guy Quote)

5. Developing ExercisesThe Role of the Sponsor • What does the sponsor want to learn or demonstrate? • What does the sponsor want the participants to learn or demonstrate? • How can the exercise best assure that the sponsor’s goals are met? • What information must be provided by the sponsor? • What information will be gathered for the sponsor?

6. Developing ExercisesSpecifying Objectives • Educational and Training Objectives • Teach or train new tasks and procedures • Reinforce previous training and education • Evaluate training and education • Research and Procedural Objectives • Develop new strategies, plans, procedures • Test execution of strategies, plans, procedures • Identify issues and gaps in current strategies, plans, procedures • Build consensus for strategies, plans, procedures

7. Full Scale or “Live” Exercise Command Post Exercise Tabletop Exercises Education Training Awareness Developing Exercises Scope of Activities

8. Developing ExercisesTabletop Exercises • Normally very low cost • Anyplace, anytime • Small number of participants • Could be for any type • of objective • Could be the first phase • of a larger exercise “One step that any organization can take is to reach out to other public and private entities in its region to conduct joint tabletop exercises.” ( Andrews, 2003)

9. Developing ExercisesCommand Post Exercises • Many organizations, not many people • Frequently examines existing or new procedures • Also could be part of an exercise “buildup” • More costs, more disruption to regular activities

10. Developing ExercisesFull Scale Exercises • Highest cost • Most people involved • Inter-agency, inter-governmental, inter-sector • Occasional (but required) • Impressions and perceptions count “That’s why the most comprehensive cyberpreparedness exercises bring together people from different, interdependent sectors and government agencies and include practicing how information will be shared.” Dr. Craig Koerner, Naval War College

11. Developing ExercisesIdentifying Participants • Organization(s) • Individuals • Who is essential? • Controllers

12. Developing ExercisesDeveloping the Scenario • The Scenario • A situation into which participants are placed that requires them to make decisions • Scenario-related information • Who and what will decisions affect? • What operational information is required? • How will the scenario be changed or updated? “The scenario can have a significant, if not overwhelming effect on the decisions players are able to make.” (Perla, 1990)

13. Developing ExercisesThe Role of Controllers • Monitor participant actions • Assess interactions • Inform participants about outcomes

14. Developing ExercisesUsing Models • Models can have several purposes • Provide inputs to the exercise • Keep the exercise moving • Replicate realistic organizations, events or functions • Examples • Physical or logical environment • Functional activities (logistics, intelligence) • Sensors • Command and control • Weapons

15. Developing ExercisesTesting and Validation • Model, data, and scenario validation • Play testing • Pre-play • Final Rules

16. Exercise Case Study: Eligible Receiver • “The eye-opener exercise” • Live cyberattacks involved • DOD focused and directed • No notice to “participants” • Key lesson learned: DOD networks are highly vulnerable • Led to the formation of Joint Task Force Computer Network Defense

17. Exercise Case Study: Black Ice • Focused on regional CIP in preparation for 2002 Winter Olympics, co-sponsored by Utah Dept. of Public Safety, US DOE Office of CIP, Utah Olympic Public Safety Command • Tabletop exercise • Used to surface issues, develop and implement an action plan for “disaster resistant Olympics” • Key lessons learned in understanding interdependencies, communication, coordination, and resource allocation

18. Exercise Case Study: Black Demon • US Air Force exercise focused on internal networks and operators • Used to evaluate detection, response, recovery procedures • Live and simulated (range) play • Validated operational procedures, and gathered best practices

19. Exercise Case Study: Blue Cascades • Pacific Northwest critical infrastructure owners with federal, state, local governments (US and Canada) • Tabletop exercise • Physical attacks (in the scenario) led to IT failures • Key lessons learned: • Number/degree of interdependencies unknown • Regional and US/Canada coordination lacking • Unanticipated loss of communications • No mechanism for cross-border analysis and reporting • Roles, missions, role of law enforcement not understood

20. Exercise Case Study: Dark Screen • Local/regional exercise involving federal, state, local govt., industry, academia, military • “Congressionally” directed • Three Phases • Tabletop • Lessons learned implementation • Live exercise • Key lessons learned: Start small and build, broadest participation is best, many information gaps exist

21. Exercise Case Study: Livewire • Department of Homeland Security and Dartmouth University sponsored/run exercise • Simulated attacks (physical and cyber) • Focus on banking and financial sector, with other sector involvement • Government performance “certainly a B+, better than my personal expectations” – Amit Yoran • Key Lessons Learned: inter-sector coordination and information sharing need improvement

22. Other Views of Exercises • How do exercises affect industry ? • Participation • Scope (number of participants) • Business Impact • Repetition • Cost • Who pays ? • Overhead & overtime • Interrelated sectors

23. Closing Thoughts • Education, training, and awareness are valuable countermeasures, but exercises are where “the rubber meets the road” • “If you’ve never been under mass fire and suddenly you are, the odds are that your brain will shut down and you’ll do everything wrong.” - Stephen Northcutt, SANS Institute. Graphic courtesy of US Naval Postgraduate School, Winners of the 2002 DOD Cyber Defense Exercise Downloaded from: www.nps.navy.mil/PAO/Internal/ Cyber_Defense.htm

24. References • “The Art of Wargaming”, Peter P. Perla, Naval Institute Press, Annapolis, MD, 1990. • “How can information exchange be enhanced”, Richard Andrews, Security Management, vol. 47/6, pg. 162. Arlington, VA, 2003. • “More than a game”, Deborah Padcliff, Computerworld, vol. 36/37, September 2002. • “Blue Cascades” Final Report, Pacific Northwest Economic Region, 18 July 2002. • “Infrastructure Interdependencies Tabletop Exercise: Summary of Key Issues and Actions to Date”, Paula Scalingi, DOE Office of CIP, May 2001 • “Black demon tests tactics, improves network defense”, Dom Cardonita, HQ AIA/PA, Lackland AFB, Texas, Summer 2002. • “Dark Screen: A Cyber Security Exercise for San Antonio/Bexar County. Final Report”, Gregory B. White, University of Texas – San Antonio, 26 September 2003. • “Simulated terrorist Attack Exposes Problems”, Ted Bridis, Associated Press, downloaded from http://www.informationweek.com, 25 November 2003. • “Cyberexercises”, Seth Cowand, University of Texas-San Antonio, unpublished manuscript, December 2003. • “Current Issues in US Homeland Security and Critical Infrastructure Protection”, Cristin L. Flynn, MCI, Inc., briefing at National Defense University, November 6, 2003