1 / 36

Money$ec Evolved

Money$ec Evolved. Wherein not everything has a tidy baseball analogy. Jared Pfost Chief Executive Officer Third Defense. Brian Keefer Security Architect Leading SaaS Security Company. Recap. Last year we applied baseball “SABRmetrics” to InfoSec We spent some time in the real world

duncan
Download Presentation

Money$ec Evolved

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Money$ec Evolved • Wherein not everything has a tidy baseball analogy Jared Pfost Chief Executive Officer Third Defense Brian Keefer Security Architect Leading SaaS Security Company

  2. Recap • Last year we applied baseball “SABRmetrics” to InfoSec • We spent some time in the real world • Oh yeah, some guy named Brad was in a movie

  3. In case you missed it How Analytics Changed Baseball

  4. Oakland A’s • Teams bid for players in Free Agent market • Start of 2002 A’s had payroll ~$40M* • NY Yankees payroll ~$126M* • So poor teams have no shot at winning, right? *From “Moneyball”

  5. 1999-2001 *Estimate from baseball-reference.com

  6. Billy Beane • GM Billy Beane defied convention • i.e. he didn’t follow “best practices” • made data-drive decisions • Hired Paul DePodesta

  7. Traditional baseball • Talent is evaluated by scouts • Scouts are usually washed-up players • i.e. “Industry veterans” or “experts” • Value statements are largely subjective

  8. Next-gen Baseball • Started in 1977 • Bill James wanted to see what influenced game outcome • Realized stats created in 1859 didn’t properly attribute events

  9. Key lessons • Don’t make emotional decisions • At least recognize your bias • Collect the “right” data • Look for correlations • Set reasonable criteria for success • Don’t overspend

  10. This Applies to InfoSec

  11. Problem statement • Every organization is competing with attackers • Most don’t have Fortune 50 budget • How can you be effective?

  12. Conventional “wisdom” • “Everyone knows” that you need • Firewall • Anti-virus • Change passwords frequently • Prohibit social networking • Etc.

  13. Do they work? • Port 80 goes through the firewall • Anti-virus misses custom malware • Stolen passwords used quickly • Social networking key to marketing and employee satisfaction

  14. Clearly this is not working • Do we actually want a new strategy? • What does winning look like? • How do we get started?

  15. Are You Ready To Win? Motivating Event

  16. What Does Winning Look Like? • Winning is not losing... • No unacceptable risks realized • Cheap as possible

  17. So, about that... • Started collecting info • Realized it was far from complete • Historical incident rates were meaningless • Minimal ability to measure what helps Money$ec 1.0 • 12 metrics

  18. Evolution Money$ec2.0 • Measure what’s easy • Set Targets • Justify More • Optimize Cost vs. Target

  19. Start With “Easy” • Incidents • # of High, Moderate, Annoying • Application • # of Post-production application bugs • Passwords • % passwords easily guessed • Scanned Vulnerabilities • # Patch & config vulns not mitigated per Severity Service Level

  20. Real Metrics Have Outcomes • Stats are trendy, Metrics have Winners|Losers • Measure actual performance against target • Benefits • Drives “acceptable risk” conversation with Management • Simplifies reporting e.g. are we above|below?

  21. Back To “Easy” • Scanned Vulnerabilities • # Patch & config vulns not mitigated per Severity Service Level • Sev 1 Server Vulns Mitigated within 30 days • Sev 2 within 60 days

  22. You really can do this

  23. Ooooh, shiny!

  24. Access Management % Employee termination within policy % Role/Access verification Network % critical systems monitored Moving to % of full packet capture Vendors % assessed per policy # overdue findings Employee # of duplicate incidents Change Management # emergency or unplanned changes % of changes with a regression Expand Measurement Every Metric Must Have A Target

  25. Server Patching 100 92 Percent 84 75 67 Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan Feb Optimize Cost - Target • Is target too high? Proposed Target

  26. 10 9 8 Post Worm 7 6 DoS Post Malware Post 5 4 3 2 1 1 2 3 4 5 6 7 8 9 10 Cost - Benefit - Accountability Evidence: Incidents, response performance, attack attempts Current Target Proposed Target Or http://code.google.com/p/openpert/

  27. Improve IR • Move IR out of IT? • Infections are incidents • Data is needed to evaluate controls • Knowing root-cause guides future controls and Targets

  28. Find Leading Indicators Integrate Metrics Into Root Cause Analysis

  29. Parting Thought • People implicitly decide not to measure. • Money$ec says explicitly decide when you don’t.

  30. Security Reformation? http://lifecypha.wordpress.com/ http://www.liquidmatrix.org/blog/2012/02/21/we-are-losing/

  31. Time to Share • Data you find useful to collect? • Spotted any correlations? • Proved any controls too expensive? • What communities do you participate in?

  32. Thanks! Brian Keefer b: http://rants.effu.se e: chort@effu.se t: @chort0 Jared Pfost b: http://thirddefense.wordpress.com e: jared@thirddefense.com t: @JaredPfost

  33. appendix

  34. RACI in action R – Responsible A – Accountable C – Contribute I - Informed (There can be only one “A”)

  35. 2011 VZ DBIR vs. Money$ec

  36. Device Patch & Config Monitoring

More Related