Security Measures and Metrics

1. Security Measures and Metrics Pete Lindstrom Research Director Spire Security

2. Agenda • Elements of metrics • Interlude: Four disciplines • Back to metrics • ROI/ROSI

3. Status of security • Difficult to define “good security” • Minimal difference between security and “lucky” • We don’t know how to measure success. • One incident doesn’t necessarily mean “failure”

4. Key elements of security metrics

5. Key elements of security metrics Activities: Four Disciplines People: AdminsbyDepartment Time: Hr/Day Month/Yr Costs: Salaries, Consulting HW, SW, Maint. Resources: User accts, systems, apps Building Blocks Let’s put them together…

6. Agenda • Elements of metrics • Interlude: Four disciplines • Back to metrics • ROI/ROSI

7. Process Effectiveness Metrics Process Effectiveness a.k.a. “doing things right” • Elements: • Activities • errors error rates • For example: • Accts per person • Vulns per person • Patches per person

8. Identity MANAGEMENT Trust MANAGEMENT Vulnerability MANAGEMENT Threat MANAGEMENT Security reference model 4. Monitor/detect inappropriate and/or malicious activity 2. Control sources (users/others) 3. Harden the Process/data • Harden the • Infrastructure

9. Four disciplines of security management Identity MANAGEMENT Threat MANAGEMENT Identity Validation Account Management Password Management Threat Identification Security Monitoring Incident Management INLINE Authentication User Access Control Intrusion Prevention Encryption Integrity System Access Control Policy Management Security Arch. Design Ticket Management Vulnerability Assessments Patch Management Software Security Trust MANAGEMENT Vulnerability MANAGEMENT

10. Identity management Functions • Identify users • Assign accounts/rights • Maintain identity (passwords) • Validate sessions • Authorize access

11. Vulnerability management Functions • Scan for exposures • Eliminate vulnerabilities • Remediate vulnerabilities • Mitigate vulnerabilities • Manage compliance

12. Trust management Functions • Write policies • Design security • Ensure confidentiality • Ensure integrity

13. Threat management Functions • Analyze traffic • Analyze logs • Manage incidents • Conduct forensics

14. Agenda • Elements of metrics • Interlude: Four disciplines • Back to metrics • ROI/ROSI

15. Process Effectiveness Metrics Process Effectiveness a.k.a. “doing things right” • Elements: • Activities • errors error rates • For example: • Accts per person • Vulns per person • Patches per person

16. Process effectiveness • Error rates • Identity management • Request errors • Vulnerability management • Vulnerabilities remaining • Threat management • Incident response • Trust management • Policy violations

17. Staff Productivity Metrics Staff productivity a.k.a. “people doing things” better • Elements: • People • Activities • For example: • Accts per person • Vulns per person • Patches per person

18. Staff productivity • Productivity and workload for all manual activities (activities/people) • Identity management • Requests per administrator • Account disablements per admin • Password resets per admin • Vulnerability management • Vulnerabilities resolved per administrator • Threat management • Incidents per person • Trust management • Policy changes per person

19. Cycle Time Metrics Cycle Time a.k.a. avg “time to perform activity x” • Elements: • Time • Activities • For example: • Accts per month • Vulns fixed per month • Patches per month

20. Process efficiency (cycle time) • Time/activities • Identity management • Request time • Vulnerability management • Remediation time • Threat management • Incident response time • Trust management • Policy creation time

21. Efficiency Metrics AdminsbyDepartment • Elements: • People • Activities • Time Efficiency a.k.a. “people doing things” quicker 2000 Hours per FTE • For example: • Accts/person/hr • Vulns/person/hr • Patches/person/hr

22. Cost Effectiveness Metrics AdminsbyDepartment Cost effectiveness • Elements: • People • Activities • Costs a.k.a. “people doing things” cheaper Salaries, Consulting Fees • For example: • Cost per acct • Cost per vuln fixed • Cost per patch

23. Cost effectiveness • Dollars/activities; dollars/resources; dollars/demographics • Identity management • Cost per request • Cost per password reset • Vulnerability management • Cost per vulnerability • Cost per system setting • Threat management • Cost per incident • Trust management • Cost per policy • Cost per project

24. When to use metrics • Process effectiveness • Six Sigma • Staff productivity • ROI / promotions • Cycle time • Balanced scorecard • Efficiency • ROI • Cost effectiveness • Activity-based costing • ROI/TCO

25. Business uses of security • Benchmarking (Balanced scorecard) • Baselining (Six Sigma) • Activity-based costing/Mgt • ROI • Risk management (ROSI)

26. Missing Element: RISK! • Elements: • Activities • Resources Risk Management a.k.a. “people doing things” more securely! • Four Disciplines: • Identity Mgt • Vuln Mgt • Trust Mgt • Threat Mgt

27. Risk metrics • Resources/resources; resources/demographics • Identity management • User accounts per application • Vulnerability management • Vulnerabilities per resource • Threat management • Incidents per resource • Trust management • Policies per resource

28. Risk effectiveness • Activities/activities (automated) • Identity management • Failed logins/total logins • Vulnerability management • Access denied/total access • Threat management • Incidents/events • Trust management

29. Agenda • Elements of metrics • Interlude: Four disciplines • Back to metrics • ROI/ROSI

30. Examples: Return on Investment (ROI) & Return on Security Investment (ROSI)

31. The elements of value (Loss) • ROI • IT productivity (time) • User productivity (time) …these also have ROSI value • ROSI • Legal/regulatory costs (fees/fines) • Direct revenue • Stored asset value (intellectual property, financial assets)

32. Let’s talk ROI • Keyword is efficiency • Reduced Capital Expenditures (CapEx) • Lower h/w, s/w costs • Scalability, manageability, performance • Reduced Operating Expenditures (OpEx) • Lower IT, end-user costs • (higher productivity)

33. Productivity • Where users and IT spend their time. • Time-is-money philosophy. • Often the only aspect of loss we quantify. • Basic source of ROI. • Hourly rate x hours of effort. • In order to determine the value of activities, you first have to determine what activities are performed.

34. Identity management ROI • Provisioning • New employee productivity • Automated account management • Password management • Reduced help desk time • Employee productivity • Web access control • Developer efficiency (build vs. buy)

35. Trust management ROI • Public Key Infrastructure • Managing certificates • Virtual Private Networks • Leased lines • SSL Acceleration • Hardware efficiency

36. Vulnerability management ROI • Firewalls • Reduce ACL management • Vulnerability assess/remediate • Reduce manual efforts • Patch management • Automate patching • Software quality • Reduce bug fixes

37. Threat management ROI • Antivirus • Recovery of systems • Network IDS • Reduce manual detection/forensics • Host IDS • Manual log efforts • Security Event Management • Aggregation/prioritization of work

38. Getting to ROI • Identify amount of labor allocated to individual security activities. • Identify solution and its corresponding activities. • Identify labor difference with and without solution.

39. The roots of ROSI • Our overall objective is to reduce risk. • We are relatively “new” to spending on solutions. • We often didn’t really do anything that was considered a recurring expense (I am guessing a bit here). • But, the Internet has changed all that (or at least made it apparent).

40. Return on Security Investment • Keyword: Effectiveness • Effectiveness = Reduced risk • Protecting Value and Loss • Legal/regulatory costs (fees/fines) • Direct revenue • Stored asset value (intellectual property, financial assets)

41. Legal/regulatory costs • Lawsuits: • Privacy suits • Downstream liability • Legal fees • Regulatory issues: • Regulatory fines • Remediation costs

42. Direct revenue • E-Commerce systems • Level of materiality • Seasons, cycles, forecasts drive expected losses • Some benchmarks: shrinkage; materiality (internal controls)

43. Stored asset value • Stored Value (financial assets) • Stored Knowledge (intellectual property) • Market Cap (or equivalent) – Book Value = Goodwill (intangible assets) • Some % of this Goodwill is attributable to information assets. • Professional services – higher percentage • Contract manufacturing or retail - lower

44. Determining loss • No physical goods • Ubiquitous supply • Full asset value is not necessarily lost • Look at loss in other ways: • Type of loss • For each application/system

45. Types of losses • How much value would be lost under the following conditions (for each app/dataset)? • Information-centric loss • Modified data (Integrity) • Copied data (Confidentiality) • Deleted data (Availability) • System/App-centric loss • Resource availability (Productivity) • Resource misuse (Liability)

46. Loss potential

47. Calculating potential loss Annual Loss Expectancy = Probability x Value ALE = P x A (Insurance Industry) • Level One: Calculate overall loss potential in 5 categories. • ALE = P x L(Assets, Revenue, Fines, IT Prod, EU Prod) • Level Two: Take above and factor in types of losses. • ALE = P x (C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E)) • Level Three: Perform above for all applications/data. • ALE = P x App1(C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E))… Appn(C(A,R,F,I,E); I(A,R,F,I,E); A(A,R,F,I,E))

48. Getting to ROSI • Determines cost effectiveness of proposed solution. • Calculate losses with and without solution. • Compare the difference.

49. Agree? Disagree? Pete Lindstrom petelind@spiresecurity.com www.spiresecurity.com