Security Metrics

1. Security Metrics Spring 2005 CS996 Information Security Management Polytechnic University Michael Aiello

2. Overview • Why do we care? • What is a metric? • How do we decide which metrics to collect? • How are they collected? • Effective risk analysis through security metrics • How do security metrics make a corporation money (operational risk)? • “Compliance competition” and security ROI • POSA Example • Problems Experienced • Homework

3. Why do we care? • How do we know how “secure” an organization is? • Metrics help define “secure” • Metrics let us benchmark our security investments against other organizations • Compliance • The metrics “gathering” process often leads to identification of security inconsistencies or holes

4. Why do we care: Example • Manager asks, “Are we secure?” • Without metrics:“Well that depends on how you look at it.” • With metrics:“No doubt about it. Look at our risk score before we implemented that firewall project. It’s down 10 points. We are definitely more secure today than we were before.”

5. Why do we care: Example • Manager Asks: “Have the changes that we implemented improved our security posture?” • Without metrics:“Sure. They must have, right?” • With metrics:“Absolutely. Look at our risk score before we made the recommended changes, and now it’s down 25 points. No question, the changes reduced our security risk.”

6. Motorola CISO on Metrics • “Security experts can't measure their success without security metrics, and what can't be measured can't be effectively managed.” (William Boni, PresidentCISO, Motorola Inc. www.secmet.org)

7. What is a metric? • The National Institute of Standards and Technology (NIST) define metrics as tools designed to facilitate decision-making and improve performance and accountability through collection, analysis, and reporting of relevant performance-related data. Metrics are simply a standard or system of measurement. In this case, it is a standard for measuring security, specifically measuring an organization’s security posture. Although there are some published standards for measuring security, ideally security metrics should be adjusted and tuned to fit a specific organization or situation.

8. Examples of metrics • Total number of remote connections over a one month period (VPN, ISDN, dial-up, remote desktop) • Maximum number of concurrent remote by user • The percentage of total applications that have a contingency plan by application criticality. • Time to analyze and recommend action on a security event • Number of Linux servers at least 90% compliant with the Linux platform security standard

9. Security Metric Categories • Platform • Number of Linux servers that are compliant with EFS policy • Network • DMZ port scans • Incident • Number of hosts infected with worm XYZ • Vendor • Average security rating for vendors that touch active customer files • People • Number of terminated employees with administrator access • Industry • Number of public security incidents in sector ABC with severity score Z • Political • Hacktivism scores, amount of sites listing sector/company ABC as potential target

10. Security Metric Types • Real Time • Number of concurrent connections to VPN • Usually from incident response systems • Polled • Number of password reset requests (monthly), • Usually from SA’s or SME’s • Incident based • Number of machines infected with worm XYZ • Number of vendors suffering from infections of worm XYZ • Usually from industry intelligence/incident response/SA’s and SME’s

11. How do we decide which one’s to collect? • Policy Mining / Easy to Spot Anomalies • Risk Scoring • ROI / Vendor Evaluations • Regulatory / Cover the industry standards • “Tips” / Visionaries

12. Policy Mining Example • Policy Statement: All users who connect remotely must be uniquely authenticated. • Enforcement Mechanism: Users are required to authenticate with a username, password and securID token to gain access to the internal network. • Network Policy (VPN): A user Kerberos account must authenticate with both the Radius and securID privilege granting servers before VPN connectivity is established • Question: How do we tell if a user is uniquely authenticated? • Metric: Maximum number of remote connections by user in a month. • Metric: Maximum number of concurrent connections for a single user • Metric: Total time connected in a single month • Metric: Number of users granted remote VPN privileges • Metric: Number of securID reset requests in a given month • Metric/Alert: user connecting to VPN from different countries simultaneously

13. Risk Scoring • Metric: Maximum number of remote connections by user in a month. • Impact: 6/10 (we care about this 6/10 relative to other metrics in this policy’s risk, this score may come from SME’s/upper management/industry direction) • Risk: 20% + (10% * last month’s count) • this is where the “soft” analysis takes place

14. ROI / Vendor Evaluations • “We spent $XXXXX on 4 new application penetration testers, are our applications more secure now? Should we hire another one?” • Metrics specific to applications not pen-tested, and those that are • “We are spending $YYYY on product XYZ, is it worth it to renew the contract or should we start looking for a new solution?” • Metrics specifically surrounding product XYZ and the problem it is solving • Number of successful social engineering attacks and their impact before and after the online training seminar

15. Regulatory (doesn’t yet exist for most industries) • Baseline metrics (from Spire Security) • Number of patched machines / total • Number services running on external facing machines • Port Scanning • Incidents

16. Standards (from Spire Security) Finances Market Cap Overall Revenue/Funding level Overall ExpensesWorkforce Number of Employees Number of contractors/temps Number of locations with dedicated ITIT Spending Budgets for Operations, Maintenance, Capital EmployeesEquipment Count of Servers, appliances, databases, client PCs, Laptops, PDAsNetwork Traffic Count of flows, possible flows, actual flows, blocked flows, sessions, commands, transactionsSecurity Spending Operations, Maintenance, Capital Expenditures, Number of Security FTEs

17. Standards contd. Identity Management Management budget Management FTEs Total User Repositories Total User Accounts Count of user accounts created, accounts modified, password resets, accounts disabled/deleted, accounts evaluatedAuthentication Events Number of failed authentications Vulnerability Management Spending Number of servers/applications/PCs scanned Number of Vulnerability Management FTEs Count of open ports, known vulnerabilities, patches, configuration changesTrust Management spending Count of Trust Management FTEs, policies written, certificates issued, signed documents, encrypted documentsThreat Management Spending Count of Threat Management FTEs, alerts, compromised systems

18. “Tips” / Visionaries • Investigations/Government/Regulator may ask information security to “monitor” specific activities • A “visionary” (author/upper management/consultant) will come up with a new/derived metric to collect in order to report on a new phenomenon

19. How are metrics collected? • Categorize and define the metric and its owner • Determine and document metric source • Automated • database connection • Script file output • Manual • Email polling • Form entry • Manual file updating • Report analysis/research

20. How are metrics collected? • Define/document collection process for each metric • A pull replication query mirrors the critical IDS alerts from server ABC database BCA to the metrics collection server DEF database BCA. DEF then sums and categorizes the alerts. The final counts are archived in table QRS in database BCA on server DEF. • Joe runs a stored procedure on server XYS database YZD which he manually correlates with Radius logs aging over the past 3 months. The report is then stored on share ABCD and Joe sends an email to Sally indicating the metric is updated. Sally then enters the metric information into the metric collection database using the form at URLQYZX

21. Effective risk analysis through security metrics • How do we make decisions based on the metrics now that we have them? • Metrics which are collected should match high impact risk items. (only spend money collecting those with high risk scores)

22. Risk Breakdown Example • Risk Measurement: Federation information security risk score (akin to homeland security colors extremely vague, policy generally shouldn’t be created based on such high indicators) • Risk Components: Network, Incident, Vendor, People, Industry, Political • Subcomponent: Federation-Global-Network-Trading-FTSE • Metric inventory for subcomponent: ID4786(A),ID2235(B),ID8674(C) • Subcomponent risk score calculation: • 50%(A*(∆last 4 months(B))) + (50% * C’s rolling average) • Security risk analysts and SMEs create score weightings

23. This is complicated and expensive, why do we do it this way? • As people, we are generally bad at concentrating on more than 7 factors/metrics/indicators at a time • Risk scoring lets us define and objectively monitor the “big picture” information security view • Correlations • Alerting / “Smarter” automated responses

24. How do security metrics make a corporation money (operational risk)? • Legislation (Basel II) says “you have to withhold 15% of last year’s revenue unless you can prove that you have mitigated your risk” • Metrics are your proof, risk scores are your slice description of the “state of the union” • In general, the less money you have to withhold/spend on insurance, the more money you make

25. “Compliance competition” and security ROI • The faster/better you can prove compliance the higher the bar is for your competition. • Reports and systems that are easy for auditors to use. • Meaningful and provable risk scores with metrics • Higher bar for competition means they spend more time/money/effort complying • Metrics give us a way of measuring security ROI • Pretend we are auditors evaluating our own organization: now we can measure our security posture before and after implementing a solution

26. POSA ExamplePolytechnic University CS996 Information Security Management(example POSA system here) • Quick reminder of system CFAC 4 Sale & user information 8 Complete transaction 5 Y/N POSA 1 Sale information 7 Complete Trans. Register 6 Y/N 2 Display Sale Info 3 User CC information USER

27. PSOA Metrics • After doing thorough risk analysis and identifying Assets/Threats/Vulnerabilities assume the following combinations are deemed the most important:

28. Ask the question, do we have policy and guidelines to mitigate/monitor these combinations? • If not, create them • In our case, assume we do • Mine these policies for potential metrics

29. Risk Measurement: Customer Credit Card Privacy Score • Risk Components: Network, Incident, Vendor, People, Industry • Subcomponent: Firmwide-Platform-Database-Internal-Access Control • Metric inventory for subcomponent: • (A)(Real-Time)Number of databases with more than 100 credit cards which do not store credit card numbers in an encrypted manner: • (B)(Real-Time)Number of system administrators with view level access to non-encrypted databases with more than 100 credit cards: • (C)(Real-Time)Number of system administrators who have criminal history with view level access to non-encrypted databases with more than 100 credit cards • (D)(Real-Time)Number of system administrators who have criminal history with view level access to encrypted databases with more than 100 credit cards • (E)(Monthly) Number of employees leaving the firm which have had access to non-encrypted databases with more than 100 credit cards: • (F)(Real-Time)Number of databases with encrypted credit cards • (G)(Real-Time)Number of administrators with ability to decrypt encrypted credit cards • (H)(Real-Time)Age of keys used to encrypt credit cards (days) • (I)(Manual)Industry time to break one credit card encryption (days) • (J)(Manual)% of administrators who have attended social engineering seminars • Subcomponent risk score calculation: • (0.75 * A * ((B * .1) + (C * .3))) * (0.25 * F * ( ....) )

30. How do we do the risk score calculation? • Risk = Threat * Vulnerability * Expected Loss • ALE (Average Loss Expectancy) = probability of loss * total loss potential • Asset Valuing • Productivity Value • Revenue Value • Liquid Financial Assets Value • Intellectual Property Value • Potential Loss • Confidentiality • Integrity • Availability • Productivity • Liability

31. How do we do the risk score calculation? • Measuring Risk • Manifest risk = ratio of malicious events to total events • (sessions, commands transactions) • Inherent risk = likelihood that a configuration will contribute to a compromise • Open ports and services running compared to historical vulnerabilities on those ports • Contributory risk = measure of process errors during normal course of operations that contribute to a compromise • User Account Management procedures

32. How do we do the risk score calculation? • Correlation • Expert systems, obvious correlations • Historical testing • Look at data leading up to an incident, see what changed • SME predictions/insight • Sometimes they just know • Industry trends • We don’t really care about 1024 bit key breaking in 2005, will we in 2012? • Firm specific • Explosion in certain incident types may necessitate a change in the equation

33. Optimization • After doing risk calculation, don’t spend time/money to collect metrics that don’t change the final score that much • Analyze risk score equation against reality, are we really reporting the proper state of the union? • Add “reliability” fields to metrics and weigh according to them as well. • Correlation/expert system analysis • Prioritize future metrics to be collected (find the “value” of a metric (risk of not collecting the metrics))

34. Problems Experienced • Systems unable to report highly critical metrics • System integration • SAs not willing/able to invest time setting up processes to collect metrics • Ad-Hoc requests/reports and their implication on the overall view • We didn’t do a penetration test on most of our servers, how can call our network secure? • Vague risk descriptions, Vague Metric Requests • Impossible metrics (usually external) • Number of credit card accounts compromised globally across any firm • Missing/incomplete historical data • Mistrust/inaction/devaluing because of qualitative components • Complete trust

35. References • http://www.secmet.org • http://csrc.nist.gov/publications/nistpubs/800-55/sp800-55.pdf • http://www1.netsec.net/content/securitybrief/archive/2004-09_Metrics.pdf • http://www.cert.org/octave/

36. Homework • Pretend you are a security analyst at Polytechnic. You have just had the following (highly simplified and fictitious) conversation with a senior manager. • Manager: “Another engineering school was just sued because student’s transcripts could be accessed by anyone online. How secure is our new grade transaction server?” • You: “We believe the new design is secure, however, haven’t allocated time, money and effort to post-implementation security evaluation. • Manager: “I need to show the board that we are not prone to this type of humiliation, what can you pull together for me in the next 3 months?” • Your assignment is to describe how you would structure your new metrics proposal which includes the following sections. • Description of which metrics you will be collecting (Based on risk analysis. Remember, this should be a minimal set, you only have 3 months to set this up) • A metric collection process example for one of the metrics • Suggested, simple weighting of metrics to calculate the overall risk score of the system.