Performance Assessment and Monitoring of Energy Efficiency Programs

1. Performance Assessment and Monitoring of Energy Efficiency Programs Mallika Nanduri Natural Resources Canada May 12th, 2005

2. Presentation Overview • Background & context • Overview of OEE’s performance assessment requirements • Current OEE work on performance assessment • General • Determining program impacts • Work to date

3. Background & Context • In 1997 Canada’s Auditor General recommended that NRCan’s Office of Energy Efficiency improve the performance reporting information for its programs by: • Estimating the energy and greenhouse gas (GHG) emissions impacts of its programs; linking the impacts to Canada’s Kyoto committments • From 1998 on • Focus on measurement of program outcomes - accountability for outcomes has only increased (as CC$ have increased) • Development of sound, informative metrics • Subsequent increase in measurement issues to resolve

4. OEE Performance Assessment Requirements • The OEE needs to: • Understand the extent to which it is meeting its mandate to improve energy efficiency • Incremental EE improvement • Incremental energy savings • Incremental GHG emissions reductions • Understand how OEE can better meet its mandate (program planning, redesign) • Meet its external monitoring and reporting obligations

5. Business Plan Metrics Metrics Metrics • Report to Parliament • CC Reporting • Energy Efficiency Trends in Canada Metrics Improved Program Effectiveness • Performance Assessment • Program Changes Increased Market Effects of Programs Internal Requirements Measurement • Qualitative • Quantitative • Performance Indicators • Prospective Impact Analyses • Retrospective Impact Analyses OEE Performance Assessment Cycle External Requirements

6. Key Measurement Issues • Auditor General (AG) guidelines recognize that • a program’s ability to influence global (aggregate) objectives and outcomes is limited • isolating that program influence from other factors is difficult • OEE efforts include: • Measuring program outcomes and performance • Addressing attribution through various means

7. Current Approaches • Performance indicators/metrics • Qualitative and quantitative indicators of annual program progress • Tracks key program activities, outputs, intermediate and final outcomes • E.g., # of workshops held, # of rebates issued • Used mainly for planning; internal and external reporting • Report to Parliament, State of Energy Efficiency Report, OEE Business Plans

8. Current Approaches • Prospective analyses • RIAS – for forecasting net impacts of EE regulations • Cost-benefit analysis framework • Energy models – for establishing longer-term outlooks on energy efficiency at the economy, sector, sub-sector level • Internal energy models • External work with energy modellers and analysts across Canada

9. Current Approaches • Retrospective analyses • Factorization methods for analyzing net trends in energy efficiency over time at the economy/sector level • Isolate energy efficiency improvements from other changes in the economy – changing activity levels, structural changes, weather • Collection and analysis of actual market data for regulated products • Track changing product efficiencies, sales over time • E.g., appliances • Estimate associated energy savings

10. Current Approaches • Impact and attribution studies • For determining net impacts of programs • Estimating free-ridership, spillover effects • Discrete choice-based surveys and analysis • Technologically and behaviourally realistic models (CIMS at Simon Fraser University) • Focus on market transformation-type effects of OEE programs

11. Total observed change in EE/energy use between two specific time periods. Includes changes resulting from the program, but also “non-program” changes that would have occurred even without the program. E.g., weather, energy prices, etc. Estimating Program Impacts Types of changes in EE/energy use Gross Change (Observed) The gross change in EE/energy use, net of non-program factors like weather and economic changes. While a good measure of overall program success, it does not account for free-rider effects. Gross Program Impact Change in EE/energy use solely attributable to the program, I.e., that would not have occurred without the program. Net of free-rider effects, natural EE improvements, etc. Net Program Impact (incremental)

12. Estimating Program Impacts Gross Program Impacts – (free-rider effects) + (spillover/indirect impacts) Net Program Impacts = GJ or Kwh saved • Identification and quantification of free-rider effects critical for OEE, as programs focus on changing consumer behaviour • OEE has had some progress with discrete choice methods • Can look at behavioural change more closely • What might have occurred in the absence of the program? • More insight into effects/impacts of the program on public

13. Studies to Date & Ongoing Work • 5 program specific studies to date – all based on DCM • Equipment, industry, housing, transport • More studies to come • One-Tonne Challenge • Energy Star • Industry • Upcoming federal processes involve ongoing and new work • Program review • UNFCCC reporting

14. End