1 / 49

UXO Risk Assessment Methods: Critical Review

UXO Risk Assessment Methods: Critical Review. Jacqueline MacDonald, Debra Knopman, J. R. Lockwood, Gary Cecchine, Henry Willis RAND. Briefing Outline. Need for UXO risk assessment methods Prioritization Site-specific assessment RAND review of existing methods: tasks, approach

keaton-mcintyre
Download Presentation

UXO Risk Assessment Methods: Critical Review

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. UXO Risk Assessment Methods: Critical Review Jacqueline MacDonald, Debra Knopman, J. R. Lockwood, Gary Cecchine, Henry Willis RAND

  2. Briefing Outline • Need for UXO risk assessment methods • Prioritization • Site-specific assessment • RAND review of existing methods: tasks, approach • Design features and limitations of existing methods • RAND recommendations for improving UXO risk assessment • Prioritization • Site-specific assessment

  3. Need for UXO Risk Assessment • Prioritization: mandated by Congress • Site-specific assessment • Cost differences among alternative response options are significant • Total current cost estimate of $14 billion assumes one scan of site and excavation of anomalies only • Alternative approaches proposed by regulatory agencies could cost much more • DOD needs to understand risk reduction differences among alternative approaches

  4. Alternatives for UXO Response • Surface clearance only • Scan with metal detector, and excavate each anomaly to a specific depth (e.g. 2 ft, 3 ft) • Scan, excavate each anomaly, scan bottom of hole, and excavate again if anomaly is detected • Scan and excavate anomalies, and then repeat the process two or more times • Excavate the entire site in one-foot lifts to depths of 2 ft, 4 ft, or more; sift the excavated soil to remove UXO

  5. Scan land with best available technology (at surface, one scan effort) Excavate all anomalies found to maximum depth (4 feet) Provide construction support to property developers Scan land (surface & 1ft) Excavate anomalies found Excavate entire site to 2 ft Scan land (2ft level) Excavate anomalies found Excavate entire site to 3 ft Scan land (3ft level) Excavate anomalies found Excavate entire site to 4 ft Scan land (4ft level) Excavate anomalies found Alternatives Proposed at Ft. X Army’s Preferred Approach State EPA’s Preferred Approach

  6. Cost Differences Among Ft. X Options Are Hundreds of Millions of Dollars

  7. Briefing Outline • Need for UXO risk assessment methods • Prioritization • Site-specific assessment • RAND review of existing methods • Design features and limitations of existing methods • RAND recommendations for improving UXO risk assessment • Prioritization • Site-specific assessment

  8. RAND Tasks • Client: Army Chief of Staff for Installation Management • Tasks: • Conduct a preliminary analysis of ongoing efforts in UXO risk assessment, including: • Ordnance and Explosives Cost-Effectiveness Risk Tool (OECert) • Interim Range Rule Risk Methodology (IR3M) • Ordnance and Explosives Risk Impact Analysis (very similar to IR3M) • Risk Assessment Code (RAC)

  9. Tasks, continued • Study methods used by the Department of Energy, National Aeronautics and Space Administration, and others to evaluate and measure risk of low-probability and high-consequence events. • Recommend how the Army could develop a risk assessment/risk management protocol for UXO sites.

  10. Study Approach • Develop criteria for a technically sound risk assessment, based on risk assessment literature survey and consultations with experts • Read all documentation for available methods; test software if available • Interview method developers • Evaluate extent to which each method satisfies the evaluation criteria

  11. Evaluation Criteria Are in Three Categories • Risk calculation features, e.g. • Is the logic sound? • Are assumptions reasonable? • Is the output reproducible? • Implementation features, e.g. • Is the method practical for intended use? • Are instructions clear? • Is it adaptable to different sites? • Communication features, e.g. • Can stakeholders provide input about assumptions, scenarios? • Is the method transparent?

  12. Briefing Outline • Need for UXO risk assessment methods • Prioritization • Site-specific assessment • RAND review of existing methods • Design features and limitations of existing methods • RAND recommendations for improving UXO risk assessment • Prioritization • Site-specific assessment

  13. Interim Range Rule Risk Methodolgy (IR3M) Purpose: To provide “a comprehensive process for managing, assessing, and communicating risks on former ranges” Four tools: • Explosives Safety Risk Tool • Other Constituent Risk Tool for Human Health • Ecological Risk Tool • Comparative Analysis Tool

  14. Explosives Safety Risk Tool • Provides a risk value on a dimensionless scale of A to E • The A to E value is determined from decision rules that consider three input factors: • Accessibility (A) • Overall hazard (O) • Exposure (E)

  15. Explosive Safety Risk Tool Output A O E

  16. RAND Evaluation of Tool • Considers main factors that determine explosion risks from UXO: hazard severity and likelihood of exposure • However, these factors are poorly quantified • The risk calculation features do not satisfy the RAND evaluation criteria: • Interactions among risk contributors are poorly modeled • Output can be illogical • Assumptions often are not explained • Uncertainties are not described • Output is not reproducible

  17. Hypothetical Example Illustrates IR3M Weaknesses • Conditions for hypothetical site: • Climatic conditions are such that buried UXO is unlikely to surface • Significant digging at the site is unexpected • All other conditions are such that O = 5, E = 2 • Two UXO response options: • Option 1: Excavate UXO to 0.99 ft of depth • Option 2: Excavate UXO to 1.01 ft of depth

  18. Calculate A for Options 1 and 2 Depth below land surface 1) All UXO > 10 feet 2) All UXO > 4 feet 3) All UXO > 2 feet 4) All UXO > 1 foot 5) Any UXO < 1 foot Migration /Erosion Potential 1) Very stable 2) Minor migration 3) Moderate migration 4) Significant migration 5) Highly dynamic Intrusion Level of Activity 1)Non-intrusive: surface only 2) Minor: to 1 ft 3) Moderate: to 2 ft 4) Significant: to 4 ft 5) Highly intrusive Option 1: A = 5 Option 2: A = 3

  19. IR3M Output for Options 1 and 2 Is Not Logical A = 5 A = 3 O = 5 E = 2

  20. Participants in IR3M Test Exercise Reached Similar Conclusions • IR3M was tested by two independent teams in a hypothetical site evaluation exercise • The teams found problems similar to those encountered by RAND: “Intuitively, the scores did not make sense. . . . If action removes all known UXO, the tool does not reflect a change in protectiveness and therefore there is a flaw in the tool.” “With a 5 on Overall Hazard and Accessibility, reducing exposure does not alter the overall risk. If there is one hand grenade at land surface, the overall risk is E. Thus, there is no way to reduce risk by reducing exposure.”

  21. Army Science Board (1998) Also Reached Similar Conclusions • “The existing R3M model is not sufficiently grounded or developed to accompany promulgation of the Range Rule in FY99.” • “Do not highlight the existing risk formulation in working toward promulgation of the Range Rule.”

  22. Ordnance and Explosives Cost-Effectiveness Risk Tool (OECert) • Purpose: • “The . . . OECert . . . provides the capability to estimate risks and cost associated with given levels of remediation and to prioritize among any set of sites by life cycle cost and the amount of reduction in public risk expected to be realized from the remediation process.”

  23. OECert Risk Calculation • Three methods: • Dispersed sector method (ranges, impact areas) • Localized sector method (burial pits, trenches) • Water sector method (submerged UXO) • For all methods: • Risk = (Number of Expected UXO Exposures) • x (UXO Hazard Factor)

  24. Dispersed Sector Method: UXO Hazard Factor

  25. Dispersed Sector Method: Exposure Calculation • Site is divided into homogeneous “sectors” • UXO density is assumed homogeneous in each sector • Density is calculated using statistical sampling OR based on densities of similar sites in a database of Formerly Utilized Defense Sites • Within each sector, number of exposures to UXO is calculated for 19 activities

  26. Dispersed Sector Method Considers 19 Activities

  27. Example: Camping • Total number of exposures is calculated for • hiking to campsite • setting up camp • Number of expected campers is determined from • demographic data for surrounding area • generic (nation-wide) estimates of percentage of people in each age category who camp

  28. Exposures During Hiking to Campsite • Distance of path to campsite is estimated as radius of a circle with area equivalent to area of sector • Area traversed by single individual: • traversed area = (sector area/π)1/2 x (path width) • Number of UXO exposures for single individual: • exposures = (traversed area) x (surface density)

  29. Exposures At Campsite • Assumed area of campsite = 2,500 ft2 • Assumed subsurface intrusive area = 0.75 ft2 • Exposures within campsite: • exposures = 2,500 ft2 x (surface density) + • 0.75 ft2 x (subsurface density)

  30. Total Exposures for UXO Site

  31. RAND Evaluation of OECert • Overall structure of modeling UXO explosion risk has merit (exposure probability times exposure consequences) • However, there are major limitations: • Contains many unjustified assumptions (e.g., campsite size of 2,500 ft2, camper intrusion area of 0.75 ft2) • Army staff report that output is usually 100,000 exposures • Software crashed when we tried to use it • Output is not meaningful outside OECert context • Process is not transparent • Density estimation procedures need to be improved

  32. Risk Assessment Code (RAC) • Purpose: Prioritize sites in the Formerly Utilized Defense Sites (FUDS) program based on UXO explosion risk • Two factors considered: • Hazard severity • Hazard probability

  33. Hazard Severity Score • Based on five broad categories of ordnance: • Conventional ordnance and ammunition • Pyrotechnics • Bulk high explosives (uncontainerized) • Bulk propellants • Chemical warfare materiel and radiological weapons • A score is developed for each category based on types of ordnance present • Total score is based on sum of scores for the 5 categories (maximum possible is 61)

  34. Example: Conventional Ordnance Score Largest single value is chosen

  35. Hazard Probability Score • Based on five subscores related to probability of human exposure to UXO: • UXO location (surface, subsurface, etc.) • Distance to nearest inhabited location • Number of buildings within 2-mile radius • Types of buildings (schools, residential, industrial, etc.) • Site accessibility (barriers, guards, etc.) • Site dynamics (e.g., erosion, future development) • Total score is sum of these six subscores (maximum possible is 30)

  36. RAC Score Is on 1-5 Scale

  37. RAND Evaluation of RAC • Model of estimating risk based on exposure probability and explosion severity is sound • Clear, easy-to-use instructions • Appropriate for intended use (priority setting) • Not detailed enough for site-specific assessment • Limitations • UXO density is not considered in the score • Does not address uncertainty • Does not address issues of stakeholder involvement and communication

  38. Summary: Existing Methods Do Not Satisfy Criteria • OE-CERT • RAC • IR3M • OERIA • Risk calculation features • Implementation features • Communication features = does not satisfy evaluation criteria = satisfies some criteria = satisfies all criteria

  39. Briefing Outline • Need for UXO risk assessment methods • Prioritization • Site-specific assessment • RAND review of existing methods • Design features and limitations of existing methods • RAND recommendations for improving UXO risk assessment • Prioritization • Site-specific assessment

  40. We Recommend a Two-Stage Process for Prioritization • Two stages: • Stage 1: sort sites into explosion risk “bins” • Stage 2: within bins, sort by munitions constituent risks Key design feature: • Preserves separate information about explosion, other constituent risks • Separation is necessary; attempts to combine these risks involve value judgments that should be explicit and transparent in decision-making

  41. Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Munitions constituent risk Modified RAC? Determine explosion risk 1 Explosion risk None Low Moderate High No explosion risk sites Low explosionrisk sites Moderate explosionrisk sites High explosionrisk sites 2 Determine munitionsconstituent risk for eachsite in category Munitions constituent risk Munitions constituent risk Munitions constituent risk Hazard Ranking System or Relative Risk Site Evaluation Primer Rank Rank Rank Rank Policymakers decide how to allocate resources among bins

  42. Briefing Outline • Need for UXO risk assessment methods • Prioritization • Site-specific assessment • RAND review of existing methods • Design features and limitations of existing methods • RAND recommendations for improving UXO risk assessment • Prioritization • Site-specific assessment

  43. We Searched Other Agencies for Models

  44. Example: Army Uses PRA to Assess Risk of Chemical Weapons Release Fault tree used in PRA for Tooele Chemical Agent Disposal Facility

  45. Potential Benefits of Using PRA for Explosion Risk Assessment • Widely and successfully used by other agencies and private industry • Could allow for meaningful quantitative evaluation of risks and benefits of response alternatives • Requires systematic analysis of elements of risk, which provides formal structure for decision-making and negotiations

  46. Army Science Board Made Similar Recommendation • Army Science Board reviewed IR3M in October 1998 • Army Science Board recommended that the DOD and Army: • “consider the probabilistic and step (I.e., ON or OFF) nature of explosive events and consequences” • “consider using an approach analogous to the successful formulation of design codes based on the probability of structural failure”

  47. Summary of Recommended Approach for Site-Specific Risk Assessment • Use EPA Risk Assessment Guidance for Superfund methods to assess risks of munitions constituents • Develop probabilistic risk assessment method specific to UXO to assess explosion risks • Development should be overseen by a technical advisory committee • The method should be independently peer reviewed • Template “trees” should be developed • Trees then could be modified at individual sites, with substantial stakeholder input

  48. Summary • Existing methods for UXO risk assessment do not satisfy criteria for technical credibility • New prioritization method should include a two-tier screen: • Tier 1: sort by explosion risks (using new method—possibly a modified RAC) • Tier 2: sort by constituent risks (using existing method from EPA or Defense Environmental Restoration Program) • New site-specific assessment method should • Use EPA’s RAGS for constituent risks • Use new PRA method for explosion risks

  49. “Fear of harm ought to be proportional not merely to the gravity of the harm, but also to the probability of the event.” • --Antoine Arnauld, 1660

More Related