Risk Analysis Overview Pete Jacobs, BuildingMetrics Inc. Steve Kromer, EVO Nick Hall, TecMarket Works March 28, 2006 CP

1. California Evaluation Protocols Public Workshop Risk Analysis Overview Pete Jacobs, BuildingMetrics Inc. Steve Kromer, EVO Nick Hall, TecMarket Works March 28, 2006 CPUC, San Francisco, CA

2. Objectives of Risk Analysis • Estimate uncertainty in program energy and demand savings • Identify programs with the greatest uncertainty • Identify metrics that can help allocate evaluation resources to minimize uncertainty • Identify risk components • Unit energy savings estimates • Net to gross ratios etc.

3. Issues Not Addressed • Not designed to estimate the overall uncertainty in the portfolio savings • Provides estimates of relative contribution of programs to portfolio uncertainty • Improve reliability of evaluation-produced portfolio savings estimates • Not designed to estimate the probability of specific implementers meeting goals • Improves the ability to assess if goals are met after the fact

4. Issues Not Addressed • Does not address the issue of evaluation effectiveness • Defined as the value of uncertainty reduction per dollar of evaluation cost

5. Risk Analysis Continuum • Overall Evaluation Planning • Allocate evaluation resources over the portfolio • Detailed Evaluation Plans • Refine the analysis at the program level • M&V Plans • Component of M&V plans at the site level

6. Energy Savings Model where: units = number of installations NTG = net to gross ratio EUL = effective useful life

7. Peak Demand and Gas Savings

8. Program and Portfolio Savings

9. Uncertainty Assignment • Uncertainty assigned to Units, Unit kWh, kW and therm savings, and NTG ratio • Unit (measure count) uncertainty relative to program tracking system • Uncertainty not assigned to EUL • EUL not addressed in these studies

10. Measure Uncertainty Model

11. Uncertainty Estimation Process • Measures assigned to measure groups • Unit kWh, kW and therm uncertainties assigned by measure group • Programs classified by delivery strategies • Units and NTG uncertainty assigned by delivery strategy • Monte Carlo simulation analysis used to estimate uncertainty in measure, program and portfolio savings

12. What is Monte Carlo Simulation? • Numerical technique for calculating uncertainty in a result from the uncertainty in the inputs • Start with a simple, deterministic model (Excel spreadsheet) • Randomly vary the inputs to the model within the range of the expected uncertainty • Tabulate the effect of these random variations on the results • Use software to automatically conduct thousands of trials

13. Why Monte Carlo Simulation? • Handles complex problems than can’t be easily solved analytically • Integrates with Excel spreadsheets used throughout planning process • Provides canned output graphics to quickly interpret results

14. Primary Data Sources • Avoided Cost Workbooks (ACW) from June 1, 2005 filing • IOU forecasts of units, unit savings, NTG and EUL by measure for each IOU-implemented program • ED assignment of delivery strategies by program • Weighted estimate of combination of delivery strategies used by each program • Expert opinions on parameter uncertainties

15. Res Appliances Res Appliances Recycling Res Cooling Res Duct seal and AC tune-up Res Exterior lighting Res Glazing and skylights Res Heating Res Interior lighting Res Interior screw lighting Res Lighting controls Res Opaque Shell Res Other Res Water heating Res WB and custom Res WH controls Residential Measure Groups

16. C&I Appliances C&I Cooling C&I Duct seal and AC tune-up C&I Exterior lighting C&I Food Service C&I Glazing and skylights C&I Heating C&I HVAC Controls C&I Interior lighting C&I Interior screw lighting C&I Lighting controls C&I Motors C&I Motor controls C&I Opaque Shell C&I Other C&I Process C&I Refrigeration C&I Retro-commissioning C&I Water heating C&I WB and custom Commercial and Industrial Measure Groups

17. Downstream Deemed Rebates Midstream Rebates Upstream Rebates Building Calculated Rebates Process Calculated Rebates Audits Quality Installation Appliance Early Retirement Financing New Construction Building Design Assistance Benchmarking Energy Management Services Building Commissioning Direct Install Technology Commercialization Upstream Training Downstream Training Targeted Marketing Mass Marketing Delivery Strategies

18. Measure Unit Savings Uncertainty Assumptions • Depends on whether ACW savings were DEER-based or not • kWh and therm uncertainty ranges • DEER-Based:  7% to  30% • Non-DEER:  12% to  40% • Peak demand uncertainty ranges • DEER-Based:  9% to  42% • Non-DEER:  18% to  56%

19. Typical High Uncertainty Measures • kWh and Therm • “Other” • Duct sealing and AC tune-up • Industrial processes • kW • Residential Lighting controls • Water heating controls • Residential screw-in lighting

20. Typical Low Uncertainty Measures • kWh • C&I Interior Lighting • C&I Interior Screw-in Lighting • C&I Exterior Lighting • kW • Same as above • Therm • Appliances • Water heating

21. Unit and NTG Uncertainties Assumptions • Unit uncertainties by delivery strategy ranged from  5% to  30% • NTG uncertainties by delivery strategy ranged from  10% to  40%

22. Typical High Uncertainty Delivery Strategies • High units uncertainty • Marketing • Upstream rebates • Audits • High NTG Uncertainty • Benchmarking • Upstream rebates • Audits

23. Typical Low Uncertainty Delivery Strategies • Low units uncertainty • Direct installation • Downstream rebates • Comprehensive measures • Low NTG Uncertainty • Training • Commercialization • Direct Installation

24. Distribution Type Assignment • Units, kWh/unit, kW/unit and therm/unit uncertainties used triangular distribution • Mean at peak • Min/max defines tales • Units constrained to values  1 • Net to gross ratio used uniform distribution • Defined by min and max values • 0.3  NTG  0.96

25. Model Savings Forecast • Savings Probability Distribution • Statistics • Mean • Standard Deviation

26. Sensitivity Charts

27. Risk Analysis Metrics • Relative uncertainty • The relative contribution of the uncertainty in each program to the overall portfolio uncertainty • Forecast mean compared to model mean • The percentage difference between the IOU point estimates and the mean savings predicted by the RA model • Spread • The uncertainty in the individual program estimates

28. Relative Uncertainty • Quantifies the relative risk each program presents to the overall portfolio • Also called the “contribution to variance”

29. Relative Uncertainty Portfolio Uncertainty Percent contribution of each program to portfolio uncertainty

30. Forecast Mean to Model Mean • Relative uncertainty looks at program contribution to portfolio uncertainty around RA mean estimate • This metric identifies programs where program mean value from RA and IOU point estimates are different

31. Spread • Measure of an individual program uncertainty as a percent of mean savings • Equal to twice the relative precision (at 90% confidence) • Identifies programs with high individual uncertainty that may have low relative uncertainty Spread

32. Electricity Savings and Uncertainty

33. Peak Demand Savings and Uncertainty

34. Gas Savings and Uncertainty

35. kWh Relative Uncertainty

36. kW Relative Uncertainty

37. Therm Relative Uncertainty

38. Program kWh Risk Components

39. Program kW Risk Components

40. Program Therm Risk Components

41. Lifetime kWh by Measure Group

42. Peak Demand by Measure Group

43. Lifetime therm by Measure Group

44. Measure Group kWh Risk Components

45. Measure Group kW Risk Components

46. Measure Group Gas Risk Components

47. Lessons Learned • Inconsistencies in Avoided Cost Workbooks across programs • Measure names - often different for the same technology or measure • Level of detail in program savings estimate • Program vs. end use vs. measure • Range from one line to > 200 • Interpretation of units and unit savings • Miscellaneous errors discovered

48. Lessons Learned • Uncertainty Assignments • Future emphasis on uncertainty analysis within EM&V studies will help better define values directly from measurements • Model specification • Future analysis should consider correlation across variables and biases

49. Lesson Learned • Expand analysis to include parameter models • E.g. unit kWh savings as a function of kW, operating hours and so on • Assign uncertainties to each term in parameter model