Probabilistic and Sensitivity Analysis for Decision Processes

1. Methods and Applications of Probabilistic and Sensitivity Analysis for Models Used in Decision Processes H. Christopher Frey, Ph.D. Professor Department of Civil, Construction, and Environmental Engineering North Carolina State University Raleigh, NC 27695 Currently on sabbatical as Exposure Modeling Advisor to the National Exposure Research Laboratory, U.S. Environmental Protection Agency, Research Triangle Park, NC 27701 Prepared for: Annual Meeting Society for Risk Analysis – Europe Ljubljana, Slovenia September 12, 2006

2. Outline • Why are probabilistic and sensitivity analysis needed? • Overview of methods for probabilistic analysis • Model inputs • Empirical data • Expert judgment • Model uncertainty • Scenario uncertainty • Overview of methods for sensitivity analysis • Recommendations

3. Variability and Uncertainty • Variability: refers to the certainty that • different members of a population will have different values (inter-individual variability) • values will vary over time for a given member of the population (intra-individual variability) • Uncertainty: refers to lack of knowledge regarding • True value of a fixed but unknown quantity • True population distribution for variability • Both depend on averaging time

4. Variability • Sources of Variability • Stochasticity • Periodicity, seasonality • Mixtures of subpopulations • Variation that could be explained with better models • Variation that could be reduced through control measures

5. Uncertainty • Sources of Uncertainty: • Random sampling error for a random sample of data • Measurement errors • Systematic error (bias, lack of accuracy) • Random error (imprecision) • Non-representativeness • Not a random sample, leading to bias in mean (e.g., only measured loads not typical of daily operations) • Direct monitoring versus infrequent sampling versus estimation, averaging time • Omissions • Surrogate data (analogies with similar sources) • Lack of relevant data • Problem and scenario specification • Modeling

6. Sensitivity Analysis • A study of how the variation in the outputs of a model can be attributed to, qualitatively or quantitatively, different sources of variation in model inputs. • Sensitivity analysis provides a tool to identify the inputs of greatest importance by: • Quantifying the impact of changes in input values on model output • Evaluating how variation in output values can be apportioned among model inputs • Identifying inputs contributing to best/worst outcomes of interest

7. Why are probabilistic and sensitivity analysis needed? • Strategies for answering this question: • what happens when we ignore variability, uncertainty and sensitivity? • what do decision makers want to know that motivates doing variability, uncertainty and sensitivity analysis? • what constitutes best scientific practice? • Decision makers may not care about all three, but might find at least one to be convincing (and useful)

8. When is ProbabilisticAnalysis Needed or Useful? • Consequences of poor or biased estimates are unacceptably high • A (usually conservative) screening level analysis indicates a potential concern, but carries a level of uncertainty • Determining the value of collecting additional information • Uncertainty stems from multiple sources • Significant equity issues are associated with variability • Ranking or prioritizing significance of multiple pathways, pollutants, sites, etc. • Need for identifying possible risk management strategies • Cost of remediation or intervention is high • Scientific credibility is important • Obligation to indicate what is known and how well it is known

9. When is a Probabilistic Approach Not Needed? • When a (usually conservative) screening level analysis indicates a negligible problem • When the cost of intervention is smaller than the cost of analysis • When safety is an urgent and/or obvious issue • When there is little variability or uncertainty

10. Myths: Barriers to Use of Methods • Myth: it takes more resources to do probabilistic analysis, we have deadlines, we don’t know what to do with it, let’s just go with what we have… • Hypothesis 1: poorly informed decisions based upon misleading deterministic/point estimates can be very costly, leading to a longer term and larger resource allocation to correct mistakes that could have been avoided or to find better solutions • Hypothesis 2: Probabilistic analysis helps to determine when a robust decision can be made versus when more information is needed first • Hypothesis 3: Variability, uncertainty and sensitivity analysis help identify risk management priorities, identify key weaknesses and focus limited resources to help improve estimates • Hypothesis 4: Doing probabilistic analysis actually reduces overall resource requirements, especially if it is integrated into the process of model development and applications

11. Role of Modeling in Decision-Making • Modeling should provide insight • Modeling should help inform a decision • Modeling should be in response to clearly defined objectives that are relevant to a decision.

12. Questions that Decision-Makers and Stakeholders Typically Ask • How well do we know these numbers? • What is the precision of the estimates? • Is there a systematic error (bias) in the estimates? • Are the estimates based upon measurements, modeling, or expert judgment? • How significant are differences between two alternatives? • How significant are apparent trends over time? • How effective are proposed control or management strategies? • What is the key source of uncertainty in these numbers? • How can uncertainty be reduced?

13. Application of Uncertainty to Decision Making • Risk preference • Risk averse • Risk neutral • Risk seeking • Utility theory • Benefits of quantifying uncertainty: Expected Value of Including Uncertainty • Benefits of reducing uncertainty: Expected Value of Perfect Information (and others)

14. Overview of “State of the Science” • Statistical Methods Based Upon Empirical Data • Statistical Methods Based Upon Judgment • Other Quantitative Methods • Scenario Uncertainty • Model Uncertainty • Sensitivity Analysis • Communication

15. Statistical MethodsBased Upon Empirical Data • Frequentist, classical • Statistical inference from sample data • Parametric approaches • Parameter estimation • Goodness-of-fit • Nonparametric approaches • Mixture distributions • Censored data • Dependencies, correlations, deconvolution

16. Statistical MethodsBased Upon Empirical Data • Variability and Uncertainty • Sampling distributions for parameters • Analytical solutions • Bootstrap simulation

17. Example of Benzene Emission Factor Category 3b: Nonwinter Storage Losses at a Bulk Terminal : Empirical Distribution

18. Example of Benzene Emission Factor Category 3b: Fitted Lognormal Distribution

19. Example of Benzene Emission Factor Category 3b: Confidence Interval in the CDF

20. Example of Benzene Emission Factor Category 3b: Uncertainty in the Mean 0.06 Uncertainty in mean -73% to +200%

21. Results: Fitted Lognormal Distribution, No Censoring

22. Results: Fitted Lognormal Distribution, 30% Censoring

23. Results: Fitted Lognormal Distribution, 60% Censoring

24. Results of Example Case Study: Empirical Cumulative Probability

25. Results of Example Case Study: Lognormal Distribution Representing Inter-Unit Variability

26. Results of Example Case: Uncertainty in Inter-Unit Variability

27. Results of Example Case: Uncertainty in the Mean (Basis to Develop Probabilistic Emission Inventory) Uncertainty in mean -77% to +208%

28. Mixtures of Distributions Percent of data in 50% CI: 92% Percent of data in 95% CI: 100%

29. Time Series and Uncertainty Different uncertainty ranges for different hours of day

30. Propagating Variability and Uncertainty • Analytical techniques • Exact solutions (limited applicability) • Approximate solutions • Numerical methods • Monte Carlo • Latin Hypercube Sampling • Other sampling methods (e.g., Hammersley, Importance, stochastic response surface method, Fourier Amplitude Sensitivity Test, Sobol’s method, Quasi-Monte Carlo methods, etc.)

31. Monte Carlo Simulation • Probabilistic approaches are widely used • Monte Carlo (and similar types of) simulation are widely used. • Why? • Extremely flexible • Inputs • Models • Relatively straightforward to conceptualize

32. Exhaust Gas Blowdown Boiler Feedwater Raw water Boiler Feedwater Treatment HRSG & Steam Cycle Steam Turbine Return Water Steam Gasifier Steam Shift & Regeneration Steam Gas Turbine Exhaust Cyclone Cyclone Gasification, Particulate & Ash Removal, Fines Recycle Hot Gas Desulfur- ization Coal Coal Gas Turbine Coal Handling Raw Syngas Clean Syngas Ash Gasifier Air Ash Fines Fines Sulfuric Acid Plant Tailgas Sulfuric Acid Air Air Electricity Conceptual Diagram of Probabilistic Modeling Engineering Performance and Cost Model of a New Process Technology Input Uncertainties Output Uncertainties Performance Performance Inputs Emissions Cost Inputs Cost

33. Comparison of Probabilistic and Point-Estimate Results for an IGCC System

34. Input Uncertainties Output Uncertainties Emissions Peak Ozone Chemistry Variable-Grid Urban Airshed Model (UAM-V) Local Ozone Meteorology Local NOx Initial & Boundary Conditions Local VOC

35. Probability of Exceeding NAAQS: Comparison of 1-hour and 8-hour Standards

36. Tiered Approach to Analysis • Purpose of Analyses (examples) • Screening to prioritize resources • Regulatory decision-making • Research planning • Types of Analyses • Screening level point-estimates • Sensitivity Analysis • One-Dimensional Probabilistic Analysis • Two-Dimensional Probabilistic Analysis • Non-probabilistic approaches

37. MethodsBased Upon Expert Judgment • Expert Elicitation • Heuristics and Biases • Availability • Anchoring and Adjustment • Representativeness • Others (e.g., Motivational, Expert, etc.) • Elicitation Protocols • Motivating the expert • Structuring • Conditioning • Encoding • Verification • Documentation • Individuals and Groups • When Experts Diasagree

38. An Example of Elicitation Protocols:Stanford/SRI Protocol

39. Key Ongoing Challenges • Expert Judgment vs. Data • Perception that judgment is more biased than analysis of available data • Unless data are exactly representative, they too could be biased • Statistical methods are “objective” in that the results can be reproduced by others, but this does not guarantee absence of bias • A key area for moving forward is to agree on conditions under which expert judgment is an acceptable basis for subjective probability distributions, even for rulemaking situations

40. Appropriate Use of Expert Judgment in Regulatory Decision Making • There are examples…e.g., • analysis of health effects for EPA standards • Uncertainty in benefit/cost analysis (EPA, OMB) • Probabilistic risk analysis of nuclear facilities • Key components of credible use of expert judgment: • Follow a clear and appropriate protocol for selecting experts and for elicitation • For the conditioning step, consider obtaining input via workshop, but for encoding, work individually with experts – preferably at their location • Document (explain) the basis for each judgment • Compare judgments: identify key similarities and differences • Evaluate the implications of apparent differences with respect to decision objectives – do not “combine” judgments without first doing this • Where possible, allow for iteration

41. Other Quantitative Methods • Interval Methods • Simple intervals • Probability bounds • Produce “optimally” narrow bounds – cannot be any narrower and still enclose all possible outcomes, including dependencies among inputs • Bounds can be very wide in comparison to confidence intervals

42. Other Quantitative Methods • Fuzzy methods • Representation of vagueness, rather than uncertainty • Approximate/semi-quantitative • Has been applied in many fields • Meta-analysis • Quantitatively combine, synthesize, and summarize data and results from different sources • Requires assessment of homogeneity among studies prior to combining • Produces data with larger sample sizes than the constituent inputs • Can be applied to summary data • If raw data are available, other methods may be preferred

43. Scenario Uncertainty • A need for formal methods • Creativity, brainstorming, imagination • Key dimensions (e.g., human exposure assessment) • Pollutants • Transport pathways • Exposure routes • Susceptible populations • Averaging time • Geographic extent • Time Periods • Activity Patterns • Which dimensions/combinations matter, which ones don’t? • Uncertainty associated with mis-specification of a scenario – systematic error • Scenario definition should be considered when developing and applying models

44. Model Uncertainty • Model Boundaries (related to scenario) • Simplifications • Aggregation • Exclusion • Resolution • Structure • Calibration • Validation, Partial validation • Extrapolation

45. Model Uncertainty • Methods for Dealing with Model Uncertainty • Compare alternative models, but do not combine • Weight predictions of alternative models (e.g., probability trees) • Meta-models that degenerate into alternative models (e.g., Y = a(|x-t|)n , where n determines linear/nonlinear and t determines threshold or not)

46. Weighting vs. Averaging Each Model has Equal Weight Model B Model A Probability Density Output of Interest Average of Both Models Neither Model Supports This Range of Outcomes Probability Density Output of Interest

47. Sensitivity Analysis • Objectives of Sensitivity Analysis (examples): • Help identify key sources of variability (to aid management strategy) • Critical control points? • Critical limits? • Help identify key sources of uncertainty (to prioritize additional data collection to reduce uncertainty) • What causes worst/best outcomes? • Evaluate model behavior to assist verification/validation • To assist in process of model development • Local vs. Global Sensitivity Analysis • Model Dependent vs. Model Independent Sensitivity Analysis • Applicability of methods often depends upon characteristics of a model (e.g., nonlinear, thresholds, categorical inputs, etc.)

48. Sensitivity Analysis Methods (Examples) • Nominal Range Sensitivity Analysis • Differential Sensitivity Analysis • Conditional Analysis • Correlation coefficients (sample, rank) • Linear regression (sample, rank, variety of basis functions possible) • Other regression methods • Analysis of Variance (ANOVA) • Categorical and Regression Trees (CART) (a.k.a. Hierarchical Tree-Based Regression) • Sobol’s method • Fourier Amplitude Sensitivity Test (FAST) • Mutual Information Index • Scatter Plots

49. Schematic Diagram of the Simplified Stochastic Human Exposure and Dose Simulation (SHEDS)-Pesticides Model

50. Example input Assumptions for the Simplified SHEDS-Pesticides Model: Inhalation Pathway