4 th ANZ SRA Conference Uncertainty analysis workshop

1. 4th ANZ SRA Conference Uncertainty analysis workshop Keith R Hayes CSIRO Division of Mathematical and Information Sciences 28th September 2009, Wellington

2. Overview • Part I : Introduction and linguistic uncertainty • uncertainty and its many sources • identifying and treating linguistic uncertainty • issues for qualitative risk assessment • models and quantitative risk assessment • Part II: Uncertainty analysis methods • methods for representing variability • methods for treating epistemic uncertainty • pros and cons of different approaches • Part III: Model structure uncertainty • qualitative and quantitative approaches

3. Acknowledgements • People who have helped: • Simon Barry - statistics and general mentoring • Scott Ferson - R functions for pba, helping me out of tight spots • Mark Burgman – case study material and elicitation • Petra Kuhnert – elicitation and pooling discussions • Greg Hood – R programming tips • Funding that has helped: • attendance and research partially funded by the Australian Centre of Excellence for Risk Assessment (ACERA)

4. Part I: What is uncertainty?

5. Why worry about uncertainty? • Risk and uncertainty are intimately linked • Risk occurs because the past and present can be uncertain, and the future is uncertain • Reasons why you may want to address uncertainty • perform an “honest” risk assessment (Burgman, 2005) • ensure that the wheat remain separated from the chaff • separate knowledge gaps from variability • predict, measure, learn • transparency • Reasons why you may not want to address uncertainty • takes more time and resources • paralysis through analysis • results may span decision criteria • transparency

6. What is uncertainty? • Some definitions: • a degree of ignorance (Beven, 2009), • a state of incomplete knowledge (Cullen and Frey, 1999) • insufficient information (Murray, 2002) • a departure from the unattainable state of complete determinism (Walker et al., 2003). • Large number of taxonomies and classification schemes but basically: • linguistic uncertainty, • epistemic uncertainty • variability

7. Terminology!

8. How do we represent uncertainty? • Using language • “highly certain”, “low uncertainty” • Numerically • probability • imprecise probability • Dempster-Shafer belief functions • possibility measures • ranking functions • plausibility measures • In practice • probability far and away the most popular

9. Linguistic uncertainty

10. Linguistic uncertainty • Ambiguity • arises when words have more than one meaning and it is not clear which one is meant • Context dependence • caused by a failure to specify the context in which a term is to be understood: “large scale escape” • Underspecificity • occurs when there is unwanted generality: “…in a small percentage (generally <10%) of founder fish, foreign DNA is integrated into the host genome …” • Vagueness • arises when terms allow borderline cases: “medium risk”

11. Multiple sources of linguistic uncertainty....... “When the familiarity standard for a plant or micro-organism has been satisfied such that {reasonable assurance} exists that the organism and the{other {conditions} of an introduction}are {essentially similar}to known introductions…the introduction is assumed suitable for field testing.” (National Research Council, 1989) VAGUE UNDER SPECIFIED CONTEXT DEPENDANT

12. …and it is still prevalent today. “Where uncertainty is assessed qualitatively, it is characterised by providing a relative sense of the amount and quality of evidence…This approach is used by WG III through a series of self-explanatory terms such as: high agreement, much evidence; high agreement, medium evidence; medium agreement, medium evidence; etc.” (4th synthesis report, IPPC, 2007)

13. Simple treatments for linguistic uncertainty…… • Ambiguity • clearly define all terms and clarify meaning • Context dependence • ensure context is explicit • Underspecificity • specify all available data and assumptions • provide the narrowest possible bounds on the statement in question. • Vagueness • crisp and fuzzy definitions

14. …..but its very hard to systematically eliminate!

15. Fuzzy sets (a possibility measure) to treat vagueness

16. Uncertainty in qualitative risk assessment • Linguistic uncertainty very important in qualitative RA • prevalent in the workshops, meetings and deliberations that qualitative assessment rely on • Qualitative definitions of uncertainty • undefined terms such as low, medium and high, mean different things to different people (even in the same context) • confound linguistic uncertainty with other sources of uncertainty • treatment and separation requires numerical definition • Define, elicit and manipulate • avoid bias and non-associative results • avoid systematic error associated with heuristics and dysfunctional group behaviours

17. Qualitative combination rules….

18. …can lead to bias and non-associative results

19. Variability • Variability • inherent fluctuations or differences in a quantity or process, within or between time, location or category/group • can be characterised but not reduced with additional data • probability theory good here • Sources of variability • inherent randomness • natural (demographic v environmental stochasticity) • anthropogenic • Defining the population • characterisation sensitive to the definition of the population, time frame, space domain

20. Variability: maximum SST

21. Epistemic uncertainty • Epistemic uncertainty • stems from our incomplete knowledge of the world • theoretically reducible with additional study • probability only suitable in Bayesian framework • Many sources • subjective judgement • model uncertainty • completeness and scenario uncertainty • systematic error • measurement error • implementation error

22. Subjective judgement and bias • Heuristics • “rules of thumb” that humans use to find solutions to problems • can lead to bias and errors during naïve elicitation • Unstructured group elicitation • can be prone to “dysfunctions of interactive group processes” • e.g: pressure for conformity, influence of dominant personalities • can exacerbate heuristics • Take home message • use structured elicitation to avoid systematic error wherever possible

23. Well know heuristics…. • Overconfidence • tendency to overestimate the accuracy of one’s beliefs and underestimate the uncertainty in a process. • Availability • assessors link their probability estimates to the frequency with which they can recall an event • Representativeness • assessors judge the probability that A belongs to B by how representative or similar A is to B • Anchoring • groups of assessors tends to anchor around any initial estimate and adjust their final estimate from this value irrespective of the accuracy of the initial estimate • Motivational bias • assessors can provide inaccurate or unreliable estimates because it is beneficial for them to do so

24. Overconfidence: opinions of geotechnical experts on two standard problems. The correct (measured) value for settlement depth was 1.5 cm and for height to failure was 4.9 m. The y-axis for both was rescaled so the maximum value was 1. Correct values are shown as dashed horizontal lines. The intervals show ‘minimum’ and ‘maximum’ values reported by the experts Source: Hynes and Vanmarcke (1975) in Krinitzsky (1993).

25. Motivational bias: Loss of gross world product resulting from a doubling of atmospheric CO2 by 2050, Ecologists Economists Source: Nordhaus WD (1994), Expert Opinion on Climatic Change, American Scientist, Jan/Feb: 45-51

26. How to guides to elicitation • Kynn (2008) recommends inter alia: • familiarize the expert with the elicitation process • use familiar measurements and ask questions within area of expertise • decompose elicitation into small distinct parts and check coherence with probability axioms • be specific with wording – use a frequency representation • do not provide sample numbers for expert to anchor on • ask the expert to discuss estimates and give evidence for and against • provide feedback and allow expert to re-consider

27. Elicitation example for a proportion…

28. …and for a quantity

29. Models Source: POMC SEES CDP, 2007

30. Models in risk assessment • Necessary abstractions of real world complexity • all risk assessment is predicated on a model • Model desiderata • precise, generalisable and realistic (Levins, 1993) • relevant, flexible and realistic (Pastorok et al., 2002) • impossible to maximise all three properties! • Complexity v simplicity • models range from highly abstract to highly complex • realism associated with complexity but… • complexity is not associated with accuracy

31. Modelling paradigms

32. Import risk assessment model: Canadian salmon

33. Import risk assessment model: Mango weevil

34. Part II: Uncertainty analysis methods

35. A view of the forest

36. Literature review: Who using what?

37. And the winner is…..

38. Incertitude: Interval analysis • Interval analysis • one of the simplest ways to represent (and propagate) incertitude • useful in data poor situations when only bounds can be specified • For example

39. Interval analysis: pros and cons • Pros • can handle any kind of uncertainty • makes only one assumption (the interval spans the true value) • does not rely on large amounts of data • bounds enclose the true value with certainty (for explicitly known functions) • Cons • no measure of likelihood over the interval • interval can quickly inflate and span decision criteria and if it does there is no indication of how close you are to the line • large calculations can be tricky without the aid of software • intervals only optimal if repeat parameters can be removed • division is undefined if the denominator spans 0

40. Variability: Bayesian Belief Networks A simple Bayesian Belief Network showing the relationship between forest cover, rainfall and river flow, for an environmental system related to agricultural production in Sri Lanka (from Cain 2001)

41. Naïve BBN for EPPO risk assessment

42. Alternative model structure

43. BBN pros and cons • Pros • transparent conceptual model of cause and effect relationships • simple to implement with software • graphical structure facilitates stakeholder participation and ease of communication • forces assessor to adhere to axioms of conditional probability • can provide prediction and diagnosis • Cons • nodes and states needs to be kept simple otherwise CPT become unwieldy and large • strictly acyclical: doesn’t handle feedback easily • can provide an unwarranted air of mathematical rigour and authority

44. Variability: 1° Monte Carlo Simulation

45. Variability and incertitude: 2° Monte Carlo Simulation

46. Monte Carlo Simulation: pros and cons • Pros • very popular and widely supported by software • relatively easy to implement • Cons • large computation time to accurately represent tails in complex risk models (but LHS helps) • 2° MCS mixes frequentist and subjective interpretations of probability • requires a lot of (often unwarranted) assumptions • results are very sensitive to the dependence between variates and other assumptions • most software packages can only model linear dependence but dependence relationship usually unknown (and may not be linear)

47. Implications of dependency Tail risks! Similar median risks

48. Monte Carlo Simulation Python

49. Incertitude and variability: DBA • Dependency bounds analysis • Uses Frechet limits to provide the “best” possible bounds on arithmetic operations with random variables, irrespective of the correlation or non-linear dependency between variates • Useful when there is little or no empirical information on dependence • For example widest possible bounds for sum given by: • Relies on numeric approximation of CDF • see Williamson and Downs, 1990

50. DBA example