1 / 25

Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations

Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations. WEIS 0 5. Workshop on the Economics of Information Security. James R. Conrad, University of Idaho Department of Computer Science conr2286@uidaho.edu. Contents. Introduction to the Problem

toril
Download Presentation

Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Analyzing the Risks of Information Security Investments with Monte-Carlo Simulations WEIS05 Workshop on the Economics of Information Security James R. Conrad, University of Idaho Department of Computer Science conr2286@uidaho.edu

  2. Contents Introduction to the Problem The Monte-Carlo Solution Overview of Monte-Carlo Simulations Example Analysis and Critique Conclusions

  3. Introduction to the Problem • An information security investment may need to compete for resources with other business opportunities • But many information security models rely upon experts’ estimates • And the experts’ estimates may include significant uncertainty • How can the analyst communicate an opportunity when so much is uncertain?

  4. Monte-Carlo Solution • Apply the Monte-Carlo technique to simulate and express uncertainty in information security models • This is not a new model --- this is an enhancement of existing models • While less common in the Computer Science discipline, many financial decision makers are already familiar with the Monte-Carlo approach

  5. Monte-Carlo Simulations • Specify uncertainty in probability distributions • Monte-Carlo engine samples distributions • Engine executes the security model once for each of several thousand iterations • Monte-Carlo engine captures and collects the result of each iteration distributions engine model results

  6. Monte-Carlo Simulations • Engine simulates uncertainty in the model parameters • Model continues to operate with discrete values • Extra complexity largely confined to the Monte-Carlo engine • Results can be charted as probability distributions distributions engine model results

  7. Monte-Carlo Example • Based upon Longstaff et al’s example appearing in “Are we Forgetting the Risks of Information Technology?” of IEEE Computer, December 2000 • Simulates the benefit/cost ratio of a proposed infosec investment for a financial enterprise • Modeling parameters are similar to Longstaff’s example with an added complication… • …The experts don’t agree!

  8. intrusion rates Original (pre-Monte-Carlo) Parameters & Model other parameters Intrusion Rate Parameters r1 2 Simulated annual intrusion count w/o investment e 5.00E-01 Effectiveness of investment r2 =r1*e Annual intrusion count with investment Other Parameters p1 =r1/365 Daily probability of intrusion w/o investment p2 =r2/365 Daily probability of intrusion with investment X $20,000,000,000,000 Asset value y1 $100,000,000 Cost of software assurance w/o investment y2 $200,000,000 Cost of software assurance with investment z1 1.00% Losses w/o investment z2 0.50% Losses with investment Model Calculations d1 =p1*z1 Calc damage w/o investment d2 =p2*z2 Calc damage with investment D =y2-y1 Calc cost to provide software assurance with investment d =d1-d2 Calc percentage of losses prevented by investment b =d*X-D Calc net benefit of investment bcr =b/D Calc benefit/cost ratio for investment (bcr=7.22) model benefit/cost ratio, bcr

  9. Uncertainty in the Revised Example • Consider a case in which the experts don’t agree upon an single value estimate for the annual intrusion rate (fixed at r1=2 events/year in the original problem) • The hypothetical disagreement stems from uncertainty in anticipated business practices • Experts do agree there exists a 20% chance that business practices will change in a way that will raise the intrusion rate to 20 events/year and an 80% chance that those practices will remain unchanged

  10. Uncertainty in the Revised Parameters • Model variability of optimistic intrusion rate as a Poisson process (for purposes of this example), ro=randpoisson(2) • Model variability of pessimistic intrusion rate as a Poisson process, rp=randpoisson(20) • Model uncertainty of anticipated business conditions by choosing the optimistic rate 80% of the time and the pessimistic rate 20% of the time using randdiscrete(0.80,0.20,ro,rp) • Variability refers to a truly random process • Uncertainty refers to the experts’ inability to anticipate future business conditions

  11. intrusion rates Revised Params & Model other parameters Intrusion Rate Parameters ro =randpoisson(2) Optimistic annual intrusion count w/o investment rp =randpoisson(20) Pessimistic annual intrusion count w/o investment r1 =randdiscrete(0.8,0.2,ro,rp) 80% Chance of ro. 20% Chance of rp. e 5.00E-01 Effectiveness of investment r2 =r1*e Annual intrusion count with investment Other Parameters p1 =r1/365 Daily probability of intrusion w/o investment p2 =r2/365 Daily probability of intrusion with investment X $20,000,000,000,000 Asset value y1 $100,000,000 Cost of software assurance w/o investment y2 $200,000,000 Cost of software assurance with investment z1 1.00% Losses w/o investment z2 0.50% Losses with investment Model Calculations d1 =p1*z1 Calc damage w/o investment d2 =p2*z2 Calc damage with investment D =y2-y1 Calc cost to provide software assurance with investment d =d1-d2 Calc percentage of losses prevented by investment b =d*X-D Calc net benefit of investment bcr =b/D Calc benefit/cost ratio for investment model benefit/cost ratio, bcr

  12. Simulation of Revised Example • randpoisson() and randdiscrete() sample the probability distributions in each iteration of the simulation • The Monte-Carlo engine recalculates the model for each iteration and captures the results (bcr) • The Monte-Carlo engine charts the captured simulation results (next slide)

  13. Simulation Results

  14. Why not use a weighted average of r1 and r2? • Why doesn’t the revised model simply compute a weighted average of the two possible intrusion rates? • r1 = randpoisson(2)*0.8+randpoisson(20)*0.2 • The randdiscrete() simulation preserves the bimodal nature of the experts’ disagreement. • Any attempt to “average away” that uncertainty conceals the truth: The experts don’t agree.

  15. Analysis • The results reflect the experts’ strong preference for the optimistic intrusion rate in which the benefit/cost ratio remains unchanged at 7.22. Risk-tolerant decision makers might manage to this value. • The mean value lies at 22 between the two modes. • The results also reflect a second mode at about 81 along with a 10% chance of the benefit/cost ratio exceeding 81. Risk-adverse decision makers might manage to this value to avoid a catastrophe “on their watch.”

  16. Critique • But are real experts willing to provide even more estimates? • The author’s industry experience with Monte-Carlo models is that many experts are relieved to disclose the uncertainty they know to be in their estimates • What real experts truly dislike is being held accountable to an expected value they know is merely representative of the possibilities

  17. Additional Critique • Given a tool to express uncertainty as probability distributions, which distributions closely model the empirical evidence? • How to extend the Monte-Carlo approach to graphical models?

  18. Conclusions • Monte-Carlo techniques offer an approach to simulate uncertainty in expert estimates • Enables the use of probability distributions for model parameters and forecast results • The Monte-Carlo engine simulates random variables, allowing a security model to continue to manipulate discrete values with only minimal changes • May be particularly useful for visualizing the potential of an extreme event, the unlikely possibility of a catastrophic outcome

  19. Questions and Optional Slides

  20. Why Poisson Distribution? • The example problem uses a Poisson process to approximate intrusion attempts • If and/or when the Poisson process usefully reflects empirical intrusion attempts is an open question • Review: Models the number of events occurring during a specified time interval for a Poisson process • Review: Continuous opportunity for independent events to occur • Review: Long-term rate is constant • Review: Used to model lightening strikes in a storm

  21. Correlated Parameters • “Every iteration of a… model must be a scenario that could physically occur.” -- Vose. • The parameters must “make sense” to the security model! • One correlated parameter can usually be expressed as a function (relation) of another. • Consider r1 and r2 in the example. These are likely related which is why r2 is calculated as a function of r1. • If the relationship (e) between r1 and r2 is also uncertain, this too can be simulated.

  22. Variability and Uncertainty • Yes, this example lumped (simulated) variability and uncertainty together for simplicity • Vose (Risk Analysis, 2000) offers an excellent treatment of this subject for those who need to keep them separated

  23. Partitioning • Yes, partitioning is an alternative technique • The Monte-Carlo technique might be viewed as an automated approach to partitioning • …and the Monte-Carlo technique avoids the subjective choice of partition boundaries • …and the Monte-Carlo technique has commercial tool support for systems-level models.

  24. Commercial Tools • Yes, commercial off-the-shelf tools are available • They are most useful for systems-level security models. • They are less useful for low-level combinatorics security models • Search for “monte carlo simulation” and pay particular attention to the “Sponsored Links”

  25. Performance • The author’s industry experience includes Monte-Carlo simulations using “hundreds” of random distribution parameters • Yes, they required several hours to run… • In 1997! • My computer is more than 10X faster today. • Simulation multiplies model complexity by n, the number of iterations. A simulation of an O(m2) model becomes n*O(m2). • Opportunities for parallel approaches when n cannot be ignored.

More Related