Workforce Initiatives within Cyber Test & Evaluation (T&E)

1. Workforce Initiatives within Cyber Test & Evaluation (T&E) Keith Jordan, NAVAIR Digital Office Jaimie Reiff, OUSD (DT&E) Dr. Jonathan Harris, NAWCTSD Orlando

2. BLUF “Cybersecurity is the ultimate team sport. There is not one single entity that has all the answers.” –ADM Michael Rogers

3. Current Status of the Cybersecurity Workforce Recent Headlines • “Global Cybersecurity Workforce Shortage to Reach 1.8 Million as Threats Loom Larger and Stakes Rise Higher” • “The Cybersecurity Talent Gap Is An Industry Crisis” • “U.S. Cyber Workforce Shortage Worsening, Agencies Tell President” https://www.isc2.org/News-and-Events/Press-Room/Posts/2017/06/07/2017-06-07-Workforce-Shortage https://www.forbes.com/sites/forbestechcouncil/2018/08/09/the-cybersecurity-talent-gap-is-an-industry-crisis/#7d25ecefa6b3 https://www.meritalk.com/articles/u-s-cyber-workforce-shortage-worsening-agencies-tell-president/ “DOD struggles to hire and retain cybersecurity personnel, particularly those with weapon systems cybersecurity expertise” – GAO Report on Weapon System Cybersecurity

4. Key Training Characteristics • Training needs to be engaging • Continual skill development is critical in cybersecurity • Industry training can be prohibitively expensive • Theory-based training is not enough • Closed loop environment necessary to develop/train on custom TTPs • Cyber T&E community needs to build coalitions just like others have Workforce Development and Retention are vital to the DoD’s cyber success

5. USS SECURE / National Cyber Range Complex (NCRC)Capture the Flag Dr. Jonathan Harris

6. Capture the Flag (CTF) Goals • Provide a training environment that: • The cyber community can get hands-on time vice just theory-based training • Is representative of the hardware, operating systems, applications and configurations found on US DoD systems. • All Red Team operators (including all penetration testing teams and CPTs) can exercise their tactics, techniques and procedures (TTPs) including all necessary tools. • Is both fun and challenging CTF combines distributed cyber environment of USS SECURE, large scale cyber training expertise of NCRC, and training science of NAWCTSD

7. Unclassified // For Official Use Only Sample CTF Architecture National Cyber Range (NCR) Participant Sites Aircraft Ship HQ Building CTF incorporates Enterprise IT, ICS/SCADA, and Weapon System objectives

8. Intel Package • Objectives • 3 Levels of difficulty • Easy/Medium/Hard • Beginner/Journeyman/Expert • Variety of Technologies • Enterprise business systems • Air Gapped ICS/SCADA control systems • Avionics buses • Open Source Recon / Previous campaigns • Resume, Tech Manual Procedure, Network Diagrams • Pseudo code and API • Pre-deployed attacks

9. CTF Format On Demand(pilot upcoming) Hot Wash is intended to provide understanding how each flag can be accomplished and opportunity for Q&A with NCR Developers as well as other teams

10. Cyber Table Top (CTT) High Level Overview Jaimie Reiff OUSD R&E Cyber Developmental Test & Evaluation

11. Bottom Line Up Front • You can’t test to 100% • What are the significant vulnerabilities? • What are the acceptable risks? • How do you develop a plan? Some programs find it difficult and confusing to negotiate the policy and processes for developing their requirements and strategy for cyber T&E The Cyber Table Top (CTT) Wargame Exercise is one way to identify credible vulnerabilities and develop actionable requirements that can be used to design efficient T&E CTTs are mission based cyber risk assessments that align to the NIST 800-30 Risk Assessment Guide and can inform each step of the Risk Management Framework in addition to cyber T&E

12. Mission-Based Cybersecurity Risk Assessment (MBCRA) • A process of identifying, estimating, assessing and prioritizing risks based on impacts to DoD operational missions resulting from cyber effects on the system(s) being employed • Informs RMF Steps 1-5 AND informs Cybersecurity T&E planning • Activities begin in Phase 1 • Identifies mission-impacting risks to test and mitigate • Assists in focusing and prioritizing the Cybersecurity T&E effort • Several common methodologies, including Cyber Table Tops • Best practices described in Cybersecurity T&E Guidebook v2.0 Appendix X3 (FOUO Appendix)

13. The Challenge Need to ‘right size’ testing To identify what’s most important • Every communication path represents a risk • Each being vulnerable to cyber attack • Budgets cannot sustain testing every single communication path that goes in to or out of our platforms and systems • Methods need to be developed to determine what is and is not a risk • Need to consider low cost methods to assess what areas are highest risk •  Questions: • How likely is an attack to succeed in accomplishing it’s mission? • And if likely, how much of an impact will it have on the mission? • Mission-based Cyber Risk Assessments like the Cyber Table Top can provide the insight into risks and what to test

14. Cyber Table Top (CTT): What, Why? How can missions be disrupted via cyber? Candidate cyber attacks are logically plausible based on technical data provided; they are NOT proven-to-work, tested, hands-on attacks • What is a CTT? • Low technology, low cost, intellectually intensive exercise to introduce and explore the effects of cyber offensive operations on the capability of a System, SoS or FoS to execute a mission • Why is it used? • Identify potential threat vectors, risks associated with threat vectors, and potential threats from boundary systems • Categorize cyber threat consequence by likelihood and impact within the assessed mission context • Inform mitigations analysis, engineering, testing and design activities • What does it produce? • Cybersecurity risk matrices based on posited mission effects • Recommendations for actionable steps to increase resistantand resilience to cyber attacks

15. CTT: How? Seminar of two teams of SMEs with opposing missions and a Leadership Team Operational Team Mission: Step through how to use the System Under Test (SUT) within a mission scenario (from mission planning to post mission maintenance) Opposing Forces (OPFOR) Team Mission: Step through how they would identify a SUT’s cybersecurity vulnerabilities and pathways, assess access methods and effects for cyber attack missions Control Team Mission: Leadership: create, conduct, analyze, and outbrief the CTT Operational and OPFOR Teamsshare the steps in their mission plans and work through the assumptions, consequences, workarounds to successful threat attacks and how that relates to mission success Data collected during CTT feeds into post exercise analysis, cybersecurity risk matrices, and next step recommendations

16. CTTs: When? Cyber Table Top Exercises inform Design, DT&E and ATO Cyber Table Top Exercises inform response to change in threat or environment One of many mission-based cyber risk assessment methodologies aligned to National Institute of Standards (NIST) guidance for conducting information system risk assessments

17. The Four Step CTT Process Approximate Execution Time (varies depending on team experience and scope of evaluation) 30 – 60 days 3-5 days 30 – 90 days Varies Exercise Execution Post Exercise Analysis Exercise Preparation Reporting Team Mission Execution Working Meetings Refine Raw Data Select System and Scope Accept Risk Must Test Operational Mission Context Development Knowledge Elicitation Likelihood and Impact Assessment Further Analysis Logistics Personnel Planning Raw Data Collection and Review Cybersecurity Risk Matrices Mitigation/ Risk Management Reconnaissance & Cyber Mission Development

18. Step 1: Preparation Inputs/Entry Criteria Major Activities Select the team members Control Team Operational Team Cyber Opposition Force (OPFOR) Define Operational Team mission Mission Selection Scenario Development Collect reconnaissance documentation Define Cyber OPFOR mission Define products and plans 30 – 60 days Exercise Preparation • Program Office approval and support • Defined CTT objectives, classification level for conducting the CTT, deliverables, and timeline • Clearly defined subset of systems and interfaces that comprise the system under analysis • Draft Plan of Actions and Milestones

19. Step 2: Execution Inputs/Entry Criteria Major Activities Kickoff Overviews Setting the stage Team planning breakouts Exercise execution Teams collaborate Data collection and review • Missions, scenarios are developed and approved • Reconnaissance on target systems is complete and summarized • OPFOR has time to review and get clarification • Key personnel are assembled and note takers are trained 3-5 Days Exercise Execution Kickoff Breakouts Control Team Review

20. Step 3: Post Exercise Analysis Inputs/Entry Criteria Major Activities Gather data Initial analysis Normalize attacks Finalize Risk Categorize recommendations • Operational and OPFOR teams provide their initial findings to the Control Team • Raw data collected captures mission and technical impact of cyber effects for post-exercise analysis • Analysis participants identified and analysis schedule developed including leadership briefs 1-3 Months 3 Day Working Meeting 3 Day Working Meeting 3 Day Working Meeting RFI Homework Homework

21. Post Exercise Analysis Key Products • Analysis table • Documents every attack • Risk analysis • Mitigations and recommendations • Actionable information • Risk matrices • Visualize the results NIST Guide for Conducting Risk Assessments SP 800-30 Rev. 1

22. Step 4: Reporting Inputs/Entry Criteria Major Activities Prioritize recommendations Develop detailed briefing report Develop executive briefing • CTT analysis table is organized and refined with SMEs • Risk matrices are completed • Follow-on recommendations are developed • Briefing and report schedules are finalized Varies Reporting

23. CTT Benefits • Pragmatic, affordable method to implement elements of the DT/OT six cybersecurity phases • Generate actionable information on high priority/high mission impact cyber threats • Inform analysis for the system survivability Key Performance Parameter (KPP) • Define specific high-value follow-on analysis and testing to verify and quantify actual risks • Provide the Program Manager’s engineering and test team opportunities for risk reduction throughout the life cycle • Reduce the likelihood and cost of cyber vulnerability discovery during operational testing and deployment • Socialize the concepts of cybersecurity for program office and operators, bridging the gaps between systems engineering, testing, and operating

24. Summary Actionable Information – what should be tested, what can be mitigated • Can be applied to subsystems, systems, systems of systems and family of systems which can be vulnerable to exploitation individually and collectively • CTT is a low cost tool for increasing personnel (leadership and warfighter) understanding of the cyber warfare domain in a mission context • The CTT process was created to help T&E leads prioritize their limited testing resources on: • Risks of high potential impact but with some uncertainty with respect to likelihood (Must test) • Mitigation approaches to address high impact risk • Follow on analysis • Design change • Test • Accept risk with an implemented mitigation that does not require design change

25. Available Resources • CTT Facilitators • Training Registration & Calendar w/ events • CTT 101 Guidance • In depth CTT Phase guidance • DoD Cybersecurity Test and Evaluation Guidance • CTT Guidebook & other guidance • DoD Cybersecurity Test and Evaluation Guidebook • All available on CTT Intelink Site • https://intelshare.intelink.gov/sites/atlcoi/cyberTableTops/SitePages/Home.aspx

26. Questions? Jaimie.L.Reiff.civ@mail.mil