What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program (2002/2003 SPC Evaluation)

1. What a Realization: An Ex Post Impact Evaluation of a Performance-Based Program(2002/2003 SPC Evaluation) MAESTRO Evaluation Showcase July 26-27, 2006 Project Manager: Pierre Landry, SCELead Consultants: Mike Rufo, Itron; Keith Rothenburg, Southern Exposure Engineering; KEMA Inc.

2. Overview of Presentation • Evaluation Context and Background • Approach • Results and Key Findings • Recommendations

3. Context: Custom, Bidding, and SPC Type Programs • History and types • custom rebates, bidding, SPC • Goals • resource acquisition and market transformation • Common features • focus on custom efficiency measures in large C&I • encouragement of comprehensive projects • inclusion of technical engineering review • requirements for proof of project installation

4. Common Issues for Large C&I Programs • Uncertainty in savings estimates • Risk of gaming and fraud • Costs of measurement and verification • Keeping application requirements manageable yet effective • Distributing funds equitably • Minimizing free riders/maximizing net impacts • Supporting the efficiency services market

5. Background on 2002-2003 CA SPC • 3 IOUs (PG&E, SCE, SDG&E) • Targeted at Large C&I • Incentives by end use • Lighting ($0.05), HVACR ($0.14), Process/Other ($0.08) • Calculated or Measured Savings Path • Determined by utility • 90+% calculated • All projects require Site Installation Report

6. Why An Ex Post Evaluation of “Performance Contract” Program? • PY2002 & PY2003 first years SPC had independent ex post impact evaluation • Originally (’98/’99), all SPC projects required in-program M&V • Initially, 2 years of M&V, then 1 year • MT policy environment in CA (1998-2000) • Virtually no attention to ex post impact evaluation • M&V dropped as requirement in 2000 • Utilities retained right to require • Most projects done under calculated savings path • Shift back to resource acquisition focus with energy crisis and 2006-2008 program admin decision

7. Objectives Re-establish process for site-specific ex post impact evaluation Implement process for representative sample Estimate program realization rate Estimate net-to-gross Recommendations for improvement Constraints Total budget vs. program size, project complexity, # of utilities Per site budgets Limited sub-metering Took 2 years for most of sample to install projects Limited pre-metering available Self report NTGR method PY2002 SPC Evaluation Objectives and Constraints

8. Approach • Develop and implement sample design (ratio estimation) • Obtain files & savings documentation • Review applications and prepare ex post analysis plans • Conduct on-site data collection (limited metering) • Develop ex post impact estimates • Prepare detailed, site-specific documentation • Carry out quality control review • Extrapolate final ex post results to the population • Estimate free ridership • self-report method • CADMAC protocol-level battery

9. Population • Combined 2002-2003 SPC Population (apps): • Combined 2002-2003 SPC Population (kWh):

10. Sample Design • PY2002-PY2003 SPC Sample (apps)PY2002-PY2003 SPC Sample (kWh)

11. Gross Realization Rate Results

12. Gross Realization Rate Results

13. Freeridership and Net-to-Gross DEER SPC NTGR = 0.7

14. Key Findings • PY2002-2003 SPC more like a custom rebate than performance contract program • Overall realization rate reasonably high but below one • Wide range in individual realization rates • Importance of influence of largest projects • Gross savings may not be as conservative as SPC program managers intended • NTGR slightly lower than 1998-2001 period

15. Other Savings-Related Key Findings • Wide range in the quality of applications and supporting documentation • Need for increased verification and documentation of assumptions in project files • Experience and expertise levels of the reviewers vary widely • Difficulties in assessing complex industrial process projects • Limited estimation of kW peak demand savings

16. Impact-Related Recommendations • Consider targeted increases in the level of technical documentation • Consider a stronger application affidavit statement regarding savings assumptions • Further standardize the review approach and documentation requirements for recurring complex projects • Consider providing or requiring more technical support for applicants for complex projects

17. Savings-Related Recommendations (Cont.) • Improve reviewer documentation • Consider increasing: • conservatism for calculated path savings estimates • measurement for large complex projects • incentive premium for measured projects • Increase pre-installation measurement for very large projects with highly uncertain baseline conditions • Consider independent review of the SPC calculator

18. Free Ridership-Related Considerations • Increase efforts to reduce free ridership • Consider: • Higher incentive levels for higher payback measures or emerging technologies • Incorporating a payback floor • Bonus payments for first-time participants • Custom baselines for process improvements • Excluding projects that are obvious free riders

19. Evaluation-Related Recommendations • Consider shifting to ex post impact evaluations from program-year to paid-year basis • or a combination • Expand scope of ex post measurement • more measurement per site • more sites in samples • incorporate uncertainty in ex post estimates into program realization rate confidence interval • Integrate evaluator early into program process to enable pre-measurement where necessary