Security Measures & Metrics

1. Security Measures & Metrics Pete Lindstrom, CISSP Research Director Spire Security, LLC www.spiresecurity.com petelind@spiresecurity.com

2. Security Metrics II Security Metrics (Part 2): Activity-Based Security Part 2 of our mini-workshop on security metrics puts the "Four Disciplines" metrics framework into play in the real world. Pete Lindstrom discusses the hurdles that must be overcome in order to get a program off the ground. Pete highlights the data and knowledge gained from this process and describes the best ways to effectively begin your security metrics initiative. Note: this is a participatory session -- all attendees electing to attend this breakout receive a metrics worksheet to be completed prior to the session, which will be the best way to tailor what you learn to fit your own specific requirements. Learn to: • Define a process for collecting the information metrics • Differentiate the relative merits and drawbacks for data collection and analysis • Identify key insights into the metrics themselves and their surrounding processes

3. Security Metrics People: Departments Admins Time: Hr/Day Month/Yr Costs: Salaries, Consulting HW, SW, Maint. Resources: User accts, systems, apps

4. Define the Universe • Collect “Universe” Info • Enterprise Information • IT Organization Information • System Information • Be consistent across the board and continue to be consistent throughout. • This is one place where you can limit the scope of a project.

5. Enterprise Information • Market Value • Total Revenue (non-profits: Funding Level) • Total Expenses • Number of Employees • Number of Geographic Locations

6. IT Organization Information • Total Capital Budget / Expense • Total O&M Budget / Expense • Total Salary/Consulting Budget / Expense • Number of IT Employees (incl. contractors) • Security Budget

7. Survey Results I

8. System Information • Define the “trusted” network environment • Number of desktops/laptops • Number of servers • By Operating System • Number of applications • Inhouse/packaged • All other components • Databases, network components, appliances

9. Survey Results II

10. Gather Security Information • People • Time • Costs • Transactions

11. Person Information • Identify security FTEs. • Two employees that spend half their time on security equal one FTE. • Security is a collaborative effort, so expect lots of partial FTEs. • Operations • System/Network Admins • Developers • Customer Support

12. Calculate FTEs

13. Time Information • Annualize everything • Person information plus consultant time • One FTE = 2000 hours • Allocate security time to Four Disciplines. • By % of time • By hours

14. Allocated Time by % or Hours

15. Cost Measurements • Identify salaries (take time information above and apply a dollar value) • Identify capital expenses (H/W; S/W; Consult; Service) • Identify maintenance expenses (Consult; Maint Fee; Service)

16. Allocated Time = People Costs

17. Allocated Product Costs * appliances PRODUCTS: • Provision • Pwd Mgt • Authent. • Web Acc Control • SSO • FW / NIPS • Vuln Scan • Patch/Remediation • Shields • SRP • Net Monitor • IDS • SEM • Forensics • Policy Mgt • PKI • VPNs • Crypto • DRM/TOS

18. Transaction Measurements • Identity Management • Accounts created • Accounts disabled • Passwords changed • Vulnerability Management • Vulnerabilities identified • Vulnerabilities patched

19. Transactions per Hour

20. Uses of Security Metrics • Process Effectiveness • Six Sigma • Staff Productivity • ROI / promotions • Cycle Time • Balanced Scorecard • Staff Efficiency • ROI • Cost Effectiveness • Activity-based costing • ROI/TCO

21. Uses of Security Metrics • Trending – are you getting more or less efficient? • Benchmarking – are you doing better/worse than peers? • Forecasting – how many resources do I need for next year? • Decisionmaking – should I build or buy a solution?

22. Survey Results III – Security Budget Compared to IT Budget Correlation: 0.99073 Avg Sec/IT Budget: 6%

23. Survey Results III – Security Budget Compared to Devices Correlation: 0.9366 Avg $ per Device: $433

24. Survey Results III – Security Budget Compared to Employees Correlation: 0.91177 Avg $ per Employee: $413

25. Q1: Best Budget Predictor? Which metric do you think should be the best predictor for security budget? • IT Budget • Number of Devices • Number of Employees • Other

26. Q2: Best Explanation for Variance? What do you think would best explain the huge variance in numbers? • Level of risk tolerance • Costs too distributed to capture • Bad survey data – inconsistencies in answers • Some companies are good, some aren’t

27. Some Examples • Activity-based Costing • Cost Benefit – ROI • Incident Costs

28. Example – Cost to create user acct • ID Mgt: User Provisioning • Cost: Salaries – a five-person group of administrators create 2500 accounts annually. Average salary is $50k. • Cost: Product (hw, sw) – a user provisioning solution costs $100k total. (amortize over 5 years). • Cost: Maintenance - 20% (pay in year 1). Salaries 2500/5 = 500 accts/user $50,000/500 = $100/acct Product Costs $100,000/5 = $20k yr $20,000/2500 = $8/acct Maintenance $100,000/5 = $20k/yr $20,000/2500 = $8/acct $116 per user account created

29. What Is It Good For? • $116 per new user per year. • Allocate costs throughout environment. • Plan budget for new applications. • Measure/compare for cost effectiveness.

30. Survey Results IV – User Info User Accounts per FTE Correlation: 0.14474 Avg Accts per FTE: 4392

31. Survey Results IV – User Info User Events per FTE Correlation: 0.051865 Avg Events per FTE: 351

32. Survey Results IV – User Info User Repositories per FTE Correlation: 0.490393 Avg Rep per FTE: 182

33. Q3: Best User FTE Predictor? Which metric do you think should be the best predictor for user admin FTEs? • Number of User Accounts • Number of Events (adds/changes/deletes) • Number of Repositories • Other

34. Q4: Best Explanation for Variance? What do you think would best explain the huge variance in numbers? • Level of risk tolerance • FTEs too distributed to capture • Bad survey data – inconsistencies in answers • Some companies are good, some aren’t

35. Example 2: Cost/benefit for Patching • 2,000 Systems • $70/hr IT support • 1 hour to patch / 2 hours to recover • 10% likelihood of patch failure • 20% likelihood of compromise (pre-exploit)

36. Example 2: Cost/benefit for Patching • Pre-exploit, manual patching • Cost to Patch: • 2,000 x 70 = $140,000 • Fail: 10% x 2,000 x 70 = $14,000 • Total cost: $154,000 • Cost not to Patch: • 2,000 x 140 x 20% = $56,000 • Decision: Don’t Patch

37. Example 2: Cost/benefit for Patching • Post-exploit, manual patching • Increases risk of compromise to 80% • Cost to Patch: • 2,000 x 70 = $140,000 • Fail: 10% x 2,000 x 70 = $14,000 • Total cost: $154,000 • Cost not to Patch: • 2,000 x 140 x 80% = $224,000 • Decision: Patch

38. Example 2: Cost/benefit for Patching • Pre-exploit, automated patching • Assume 1 patch per month • Cost to Patch: • Software Costs = $48,000 • 1/12 of $48k = $4,000 • Fail: 10% x 2,000 x 70 = $14,000 • Total cost: $18,000 • Cost not to Patch: • 2,000 x 140 x 20% = $56,000 • Decision: Patch

39. Example 2 – Patching ROI • Compare two patch scenarios: • Manual process: $154,000 • Automated process: $18,000 • ROI: $136,000

40. Example 3: Cost of an Incident • Loss of value (inherent to the resource) • User Productivity • Stored Asset Value • Intellectual Property Value • Revenue Generation Value • Costs (associated w/ incident) • IT Productivity • Regulatory Fines • Opportunity Costs

41. Calculate User Productivity s = $2 billion • Identify organization’s annual salary expense (s) from financial statements. • Divide by number of employees (e) = avg salary (a). • Divide avg salary by 2000 = avg hourly rate (h). • Estimate % of employee base that are computer users (u). • Estimate % of time that employees use computers (t). • Estimate length of downtime (d). • Productivity Loss = s / e / h * u * t * d e = 40,000 employees a = $2b/40k = $50,000 h = $50k/2k = $25/hr u = 60% empl. users t = 75% (heavy users) d = 2 hours 40k * .6 * $25 * .75 * 2 = $900,000

42. Notes on Productivity • Steps 4 and 5 estimates could be replaced with avg number of simultaneous sessions and avg length of session. • Assumes you won’t spend more than something is worth. • The calculated number is a conservative one. • Does not account for indirect revenue generated (only includes person salary costs) • IT Productivity is calculated in a similar way but ends up being an extra allocated cost. • Can include opportunity cost as well.

43. Conclusions • Metrics are useful in a number of different situations. • Numbers/statistics are suspect without corresponding analysis and sensibility test. • Numbers will get better as the profession matures.

44. Agree? Disagree? Pete Lindstrom petelind@spiresecurity.com www.spiresecurity.com