Are Enterprise Security Risk Metrics Really Needed?

1. Are Enterprise Security Risk Metrics Really Needed? Robert Marchant Ph.D. Technical Fellow Sotera Defense Solutions

2. Information Systems Executives and Program Managers Need Security Metrics to Establish an Effective Security Budget. • [Evans 2004] Evans, Karen, Testimony before the Committee on Government Reform, Subcommittee onTechnology, Information Policy, Intergovernmental Relations, and the Census, 16 March 2004. • IRC recognizes Enterprise Security Metrics as a hard problem • INFOSEC Research Council 2005 Hard Problems List- number eight • The Goal is Metrics at Least as Good as What is Used for Program Risk Management. • The problem is real, but quantitative is a weak hypothesis. • Quantities are based on qualitative data • The problem and the methodology (framework) are organizationally agnostic. • The two most often used IT risk frameworks are virtually interchangeable. • [Kuligowski 2009] Kuligowski, C., Comparison of IT Security Standards. Masters Thesis, http://www.federalcybersecurity.org/CourseFiles/WhitePapers/ISOvNIST.pdf/

3. The IRC states that organizations must make cost/benefits decision based on data that is (at best) poorly quantified. • These unqualified decisions are often based on short term, poorly associated metrics that often leads to decision that result in poor, long term decision. • According the IRC, “One of the most insidious threats to security metrics lies in the metrics themselves. The mere existence of a metric may encourage its purveyors to over endow the significance of the metric. A common risk is that analyses may be based on spurious assumptions, inadequate models, and flawed tools, and that the metrics themselves are inherently incomplete --- often a one-dimensional projection of a multidimensional situation. Furthermore, a combination of metrics in the small (e.g., regarding specific attributes of specific components) typically do not compose into metrics in the large (e.g., regarding the enterprise as a whole).”

4. DMZ vsCross Domain Guard • Network Interface rules • Keep the good stuff in • Keep the bad stuff out • Don’t allow covert channels • CDG are monolithic (single probability) • DMZ is layered with multiple protocol breaks (conditional dependence – Def in Depth) • Java 1.7 • Situational probability

5. Systems engineering lifecycle all have iterative models, have feedback loops, and all use some form of control gates. Program risk assessment is performed throughout the life cycle. Risk metrics are normalized (dollarized)

6. Typical Risk Management Framework

7. FIPS 199/SP 800-60 CATEGORIZE Information System FIPS 200/SP 800-18 SP 800-30/SP 800-53 Define criticality/sensitivity of information system according to potential worst-case, adverse impact to mission/business. SP 800-37/SP 800-53A MONITOR Security Controls SELECT Security Controls Continuously track changes to the information system that may affect security controls and reassess control effectiveness. Select baseline security controls; apply tailoring guidance and supplement controls as needed base on risk assessment Security Life Cycle SP 800-70 SP 800-37 IMPLEMENT Security Controls AUTHORIZE Information System Implement security controls within enterprise architecture using sound systems engineering practices; apply security configuration settings Determine risk to organizational operations and assets, individuals, other organizations, and the Nation; if acceptable, authorize operation. SP 800-53A ASSESS Security Controls Determine security control effectiveness (i.e., controls implemented correctly, operating as intended, meeting security for information systems).

8. SDLC to RMF comparison

9. NIST Risk Management Framework Starting Point CATEGORIZE Information System Define criticality/sensitivity of information system according to potential worst-case, adverse impact to mission/business. MONITOR Security Controls SELECT Security Controls Continuously track changes to the information system that may affect security controls and reassess control effectiveness. Select baseline security controls; apply tailoring guidance and supplement controls as needed base on risk assessment Security Engineering and IT Risk Assessment Performed Here IMPLEMENT Security Controls AUTHORIZE Information System Implement security controls within enterprise architecture using sound systems engineering practices; apply security configuration settings Determine risk to organizational operations and assets, individuals, other organizations, and the Nation; if acceptable, authorize operation. ASSESS Security Controls Determine security control effectiveness (i.e., controls implemented correctly, operating as intended, meeting security for information systems).

10. The Quantitative Measure • Risk = Asset Value * Probability * Impact • Where • Asset value is in $s (SME based) • Probability based on threat analysis (SME based) • Impact (Vulnerability DB based or SME) • The sum of the risk is the total enterprise metric • Security Engineers evaluate the threat and modify/influence architecture/design to minimize. • Maintaining these metrics at the enterprise level would require a normalized taxonomy (or ontology) • Measure the probability? • Requires experts and security engineers

11. Quantitative Metrics Are Expensive • Better to minimize by lowering probability • [Coles-Kemp 2009] Coles-Kemp, L. (2009). The Effect of Organisational Structure and Culture on Information Security Risk Processes Administrative Science Quarterly, 17(1), 1- 25. • Enterprises change • New capabilities • Assimilation • Planned upgrades • Response to new threats • Normalized taxonomy (ontology)? • SMEs can be assembled as needed! • But make sure they are SMEs!!!! • Budget should be focused on maintaining security controls and ensuring patches are current • Can be empirically measured

12. Questions?

13. Backup Slides

14. Head of Agency(Element Head, Chief Executive, e.g. Director National Security Agency or Secretary of Defense): The executive with the ultimate responsibility for mission accomplishment and execution of business functions. • Risk Executive (an individual or a function): The Risk Executive function may be fulfilled by an individual or a group within an organization. The Risk Executive (organization or individual) is primarily a source of expertise and consultation and is usually a department or group (e.g. the technical security group or Cyber Security Group). • Chief Information Officer (CIO): The CIO ensures that information systems are acquired and information resources are managed in a manner consistent with laws, Executive Orders, directives, policies, regulations, as well as priorities established by the Element Head. • Senior Agency Information Security Officer (SAISO)/Chief Information Security Officer (CISO): A SAISO or CISO executes the CIO’s responsibilities under the Federal Information Security Management Act (FISMA) of 2002 and serves as the CIO’s liaison to the organizations Authorization Official. It is this individual who will aggregate all the organizations systems and programs FISMA reporting into a single agency report to the OMB.

15. Authorizing Official (AO): An AO is an agency or element CIO or executive of sufficient seniority to execute the decision-making and approval responsibilities for information systems authorizations to operate (called and ATO) on behalf of the Element Head. The AO assumes responsibility for operating an IS at an acceptable level of risk to the organization. • Delegated Authorizing Official (DAO): A DAO is delegated authority by an AO to carry out the same activities as an AO (e.g., authorize system operations). • Security Control Assessor (SCA): An SCA (sometimes called a certifier) is responsible for performing the evaluation (Asses Security Controls phase) of thesecurity-controls and features of an IS and determining the degree to which thesystem meets its security requirements. • Common Control Provider (CCP): A CCP is responsible for all aspects of providing common controls (i.e. the security controls from SP 800-53, modification to the SP 800-53 recommended controls and any custom controls augmenting SP 800-53). Organizations may have multiple CCPs.

16. Information Owner/Steward: An Information Owner/Steward is an organization official who “owns the data”. The IO has statutory, management, or operational authority for specific information and is responsible for establishing the policies and procedures governing its generation, collection, processing, dissemination, and disposal. • Mission/Business Owner (MBO): An MBO has operational responsibility for the mission or business processsupported by the mission/business segment or the information system. TheMBO is a key participant/stakeholder regarding system life-cycle decisions. • Information System Owner (ISO)/Program Manager (PM): An ISO (aka PM) is responsible for the overall procurement, development, integration, modification, operation, maintenance, and disposal of an information system (as well as the system components), to include development and provision of the stem’s Security Plan (SSP).

17. Information System Security Engineer (ISSE): An ISSE ensures that information-security requirements are effectivelyimplemented throughout the security architecting, design, development,configuration, and implementation processes. The ISSE coordinates his/hersecurity-related activities with ISOs, ISSOs/ISSMs, and CCPs. The ISSE also provides the definition, design, development, and deployment support to development systems as part of the system under developments systems engineering activity. • Information System Security Officer (ISSO)/Information System Security Manager (ISSM): An ISSM or ISSO is responsible for maintaining the day-to-day security posture and continuous monitoring of a system.