Decision Analysis in GoldSim

1. Decision Analysis in GoldSim Paul Black John Tauxe Ralph Perona Tom Stockton Neptune and Company, Inc. http://www.neptuneandco.com

2. Presentation Outline • Decision Analysis Basics • Background • Some greek! • Example context • Simple example in GoldSim • Add some uncertainty • Smoky Site • NTS low-level waste sites

3. Decision Analysis Overview • “Formalized common sense” • A set of tools for structuring and analyzing complex decision problems • An approach for making logical, reproducible, and defensible decisions in the face of: • Technical complexity • Uncertainty • Costs and value judgments • Multiple, competing objectives

4. Decision Analysis or…. • Multi-Attribute Utility Theory (MAUT) • Cost/benefit analysis • Multi-Criteria Decision Analysis - MCDA • Probabilistic modeling • Deterministic analysis does not support decision analysis in the face of uncertainty, which is really how all decisions are made.

5. Decision Analysis Overview • In the long run, you will be better off if you choose the alternative (decision option) that gives you the best expected outcome, given what you know or believe about future events • Expectation implies uncertainty • Maximize Expected Utility • Minimize Expected Losses

6. What you can do Decision options & reward structure What you knowUncertainties What you want Objectives Maximize FinancialReturns Minimize Health & Safety Impacts Maximize CustomerSatisfaction Decision Problems have 3 Basic Components

7. Decision Analysis – “Greek” • Event or outcome space, W, with events, w • Probability distribution, P(w) • Decision space, D, with decision options, d • Utility functions U(w, d), or Loss functions, L(w, d). Note L(w, d) = - U(w, d) • Expected Utility = SU(w,d)*P(w) • Objective is to find the decision option for which Expected Utility (EU) is Maximized

8. Roots of Decision Analysis • Decision Analysis established as an applied discipline and a field of research in the late 1960’s • Howard Raiffa (Harvard) • emphasis on decision analysis as a method with real world applications • Ron Howard (Stanford) • emphasis on influence diagrams and economic analyses in the face of uncertainty

9. Roots of Decision Analysis • Bayesian probability theory (Bayes, mid-1700s) • Utility theory (von Neumann & Morgenstern, 1947) • Bayesian Statistical Decision Theory (de Finetti, 1930s, Savage, 1954, DeGroot, 1970) • Behavioral Science (von Winterfeldt and Edwards, 1986) • Policy Analysis (Morgan and Henrion, 1990)

10. Common Application Areas • Oil and gas industry • Risk analysis (business decision risk) • Pharmaceutical and biotechnology industries • Public sector applications • Department of Defense • Environmental – moving in this direction

11. Environmental Evolution • Compliance with deterministic models • Difficult to overcome inertia and intransigence in the industry (old dogs - new tricks) • Difficult to overcome established regulations • Changes at the top-level take a long time to trickle down through Regions and States • Strong evidence of an evolutionary change • OMB, EPA SAB, EPA CREM, NUREG, SRA • Impact of changes in education system

12. Simple Example • Planning our wedding on Red Lodge Mountain • Options for the ceremony include: • Indoors in the ski chalet, d1 • On the covered deck, d2, and, • On the mountain, d3 • The decision is impacted by: • The chance of rain, Prob(rain), and • The likely temperature, Prob(temperature) • Rain and temperature are the w’s.

13. Simple Example • If it is a nice day out, we would prefer to be outside on the mountain (e.g., dry and hot) • If it pours and it is cold, we would prefer to be inside • If it rains, but it is still warm, we will use the covered deck area. • The decision needs to be made far ahead of time based on our best understanding of weather in Montana in mid-August.

14. Simple Example - Probabilities • To keep things simple we set up: • P(rain) = 0.2 • P(dry) = 1 – P(rain) = 0.8 • 3 categories for temperature • Prob(cold) = 0.2 • Prob(warm) = 0.5 • Prob(hot) = 0.3 • The probability space has 6 possible events

15. Simple Example – Utilities ($)

16. Simple Example - Calculations • E(U|Indoors) = P(rain&cold)*U(Indoors|rain&cold)+ P(rain&warm)*U(Indoors|rain&warm)+P(rain&hot)*U(Indoors|rain&hot)+P(dry&cold)*U(Indoors|dry&cold)+ P(dry&warm)*U(Indoors|dry&warm)+P(dry&hot)*U(Indoors|dry&hot) • I.e., sum the probability of each event multiplied by the utility of the decision option given each event.

17. Simple Example • E(U|Indoors) = $4,200 • E(U|Deck) = $6,300 • E(U|Outdoors) = $6,600 • Choose the Outdoors option (the decision option that provides the greatest benefit) • Depends on the input probabilities for rain and temperature • Example in GoldSim

18. Simple Example – Expansion? • No uncertainty built in – no simulation needed – no global sensitivity analysis possible • Uncertainty in the P(rain)? • Continuous temperature (probability distribution) • Value of further information? • Should we continue to track the weather forecast to get better information? • How much would it be worth if we could control the weather (just for that day)?

19. Simple Example – Expansion? • P(rain) follows a Beta distribution • Temperature follows a Normal Distribution • GoldSim model • Is this a better model? • Yes, because uncertainty is included • Allows global SA • Allows value of information to be assessed • Don’t make the decision until you have enough information to make it.

20. Expanded Example – Results? • E(U|Indoors)  (4497, 4500, 4510) • E(U|Deck)  (5790, 5850, 5900) • E(U|Outdoors)  (3930, 3990, 4050) • Choose to hold our wedding on the Deck • No overlap of ranges, so decision can probably be made without collecting more information • Uncertainty could be introduced into utilities.

21. Decision Analysis in Practice • Ten Commandments (Morgan and Henrion) • Model building • transparency, traceability, reproducibility, peer review – “simple as possible but no simpler” • Sensitivity Analysis (Model Evaluation) • Value of Information (do we need more data?) • Data collection • (Bayesian) Updating • Iterate until no value in collecting more

22. Decision Analysis in GoldSim • Decision Analysis math is basically simple, so it can be done in GoldSim. • GoldSim interface allows decision models to be developed using the 10 commandments. • However, GoldSim interface does not: • Build pure influence diagrams directly (formality issues) • Does not handle statistical modeling easily (units issue) • Does not handle Bayesian updating (need MCMC) • Does not do global SA (needs to be done externally)

23. When to use DA • When decisions are not trivial • Obvious outcome (e.g., drive to work or not) • Trivial topic (e.g., which egg to boil) • I.e., when decisions are hard or difficult, e.g., • Which house to buy (perhaps) • Business decisions (re-organization (e.g., GTG), mergers, acquisitions, contracts) • Political decisions (war, health care, immigration) • Environment (brownfields, DOE’s ALARA, NEPA)

24. So, What is Decision Analysis? • An overall approach for making decisions that is: • Rational • Logical • Reproducible (transparent and traceable) • Defensible • In the face of: • technical complexity • uncertainty, and • multiple, possibly competing, objectives. • A set of tools for structuring and analyzing complex decision problems

25. Applying Decision Analysis • Identify objectives, decision options, and events that define the decision analysis • Clearly communicate judgments about utilities (costs and value judgments), uncertainty (probabilities), and risks (EU) in an unambiguous way • Actively involve stakeholders, customers or users of the decision model at all stages of the decision analysis process (instead of only at later stages, which is more typical)

26. More Complex Example • The Smoky Site at the Nevada Test Site • Multi-disciplinary work, involved a technical team • Model development – assumptions, structure, quantification – all with complete documentation • Top-down GoldSim model focused on the Decision Analysis needs (as simple as possible but no simpler) • Results • Decision outcome

27. Smoky Site Background • The Smoky Site at the NTS was used to conduct safety shots and atmospheric tests of nuclear devices. • Long term maintenance of power lines that cross the Smoky Site are of concern for worker safety. • Current contamination levels require investigation under human health and environmental protection (DOE O 5400.5) and occupational exposure (10 CFR 835).

28. Smoky Site Decision Framework

29. Decision Options • Cost

30. Median Cost Breakdown

31. Temporal Cost Breakdown

32. The completed causeway

33. Smoky Site Concluding Notes • Credibility gained through peer review and thorough documentation (transparency, traceability) • Top-down GoldSim model focused on the Decision Analysis needs (as simple as possible but no simpler) • E.g., no fate and transport model – from a Decision Analysis perspective it was not needed • Iteratively refined problem statement and model • E.g., 5400.5 vs. 835.1 issues • Explicitly defined decision options, costs and uncertainties

34. An Example from the Nevada Test Site Area 5 Radioactive Waste Management Site • Photo courtesy NNSA/NSO

35. Even More Complex Example • Low-level Radioactive Waste Management Sites (RWMS) at the Nevada Test Site • Decision Objectives: • Optimize future disposal • Optimize closure design • Long term management • Optimize monitoring program with stopping rules

36. DOE Performance Assessments • Establish “reasonable expectation” that performance objectives are not exceeded (e.g. DOE M 435.1), in order to authorize waste disposal. • Compliance-based decision making • PAs are traditionally deterministic and “conservative”, yet there are inherent uncertainties in assumptions, parameter values, and in the models themselves. • ALARA (as low as reasonably achievable – 5400.5) offers a regulatory path to performing decision analysis instead.

37. Costs and Value Judgments • Disposal costs (material costs, depth of disposal, containerization) • Closure costs (material costs, cover thickness, institutional controls) • Management costs (maintenance, institutional controls) • ALARA costs (related to receptor population doses) • Other values (ecological, stakeholder concerns)

38. PAs and Uncertainty Sources of uncertainty in PA modeling include • conceptual model assumptions and exposure scenarios, • analytical and numerical models and their assumptions, and • model input parameters in space and time (variability and knowledge uncertainty).

39. Transport Model Uncertainties • The conceptual model of transport at the Area 5 Radioactive Waste Management Site at the Nevada Test Site includes: • upward flux of water driven by high evapotranspiration potentials, • diffusion in liquid and gaseous phases (radon), • biotic transport of contamination and materials in the near surface, and • resuspension and air dispersion.

40. GoldSim at the NTS

41. Important Results to date • GoldSim provides a forum for thorough documentation (transparency, traceability) • Explicit about decision options, costs and uncertainties • Perform sensitivity analysis on each iteration of the model • Iterate each time we collect data/information • DOE peer review has resulted in a fast track acceptance of the PA results.

42. Areas that need work • Bayesian updating as more data are collected • Accommodating statistical/regression models (empirical vs. mechanistic – units issue) • Correlation structures (also a statistical issue) • Model simplification (using sensitivity analysis, and Kalman filtering) could aid decision analysis • Explaining the importance of distribution averaging to match the spatial/temporal scale of the model/problem

43. RWMS Concluding Notes • A very large model driven by: • Historical modeling • Compliance-based regulation • Perception of the need for a “complete” model creates challenges for model simplification • ALARA will help us refine the model because of the decision analysis context • We have been fought every step of the way, overcoming each obstacle in turn. • It has been critical to have an advocate at DOE.