Stephen S. George, Ph.D. Principal Consultant Freeman, Sullivan & Co.

1. Load Impact Estimation for Demand Response ResourcesEstimate It, Measure It. Verify ItDRCC National Town Meeting on Demand ResponseJune 3, 2008 Stephen S. George, Ph.D. Principal Consultant Freeman, Sullivan & Co.

2. There are many potential uses for DR LI estimates How Will the DR Impact Estimates by Used? • How Much Will I Get In The Future? • Tomorrow • 3 years from now • How Much Load Reduction Did I Get? • Yesterday • Last year • How Much Do I Need to Pay For The Resource? • For last month’s performance • To keep the resource available Whatever the purpose, the objectives of load impact estimation are the same—accuracy, precision, practicality and transparency How best to achieve these objectives will vary with the purpose, program and circumstances for which the impact estimates will be used

3. The CPUC recently issued a decision establishing protocols for DR LI estimation • April 24, 2008 Decision Adopting Protocols for Estimating Demand Response Load Impacts • Rulemaking 07-01-041 • 150 page report describing 27 protocols and providing details on methods that can (but not must) be used • Focus on what should be provided, not on how to do the job • Focus on ex ante forecasting for resource planning, but also establishes protocols for ex post evaluation, which must be the basis for ex ante estimation • Establishes minimum requirements for LI estimation for DR resources • What must be considered prior to conducting the evaluation • What must be calculated, including uncertainty adjusted, hourly load impacts for selected day types • What must be reported, including selected statistics that allow reviewers to assess the validity of the analysis that underlies the estimates

4. DR resources can be categorized by whether or not they are tied to a callable event

5. According to the CA protocols, impact estimates are to be reported for the following day types

6. Methodology Methodology Methodology Methodology Methodology Methodology Develop Evaluation Plan Develop Evaluation Plan Develop Evaluation Plan Develop Evaluation Plan Ex Post Event Ex Post Event Ex Post Event Ex Post Event - - - - Based DR Based DR Based DRR Based DRR Ex Post Event Ex Post Event Ex Post Event - - - Based DRR Based DRR Based DRR (Protocol 1) (Protocol 1) (Protocol 1) (Protocol 1) (Protocols 4 (Protocols 4 (Protocols 4 (Protocols 4 – – – – 10) 10) 10) 10) (Protocols 4 (Protocols 4 (Protocols 4 – – – 10) 10) 10) M&V M&V M&V M&V M&V M&V M&V Es Post Non Es Post Non - - Event Event - - Determine Preliminary Determine Preliminary Determine Preliminary Determine Preliminary Ex Post Non Ex Post Non Ex Post Non Ex Post Non Ex Post Non - - - - - Event Event Event Event Event - - - - - Based DR Based DR Protocol Protocol Protocol Protocol Protocol Protocol Protocol Data Needs and Data Needs and Data Needs and Data Needs and Based DRR Based DRR Based DRR Based DRR Based DRR Protocol Protocol Protocol Protocol Protocol Protocol Protocol Persistence Persistence Persistence Persistence Persistence Persistence Persistence Protocols 11 Protocols 11 – – 16 16 Requirements Requirements Requirements Requirements Requirements Requirements Requirements Methods for Load Impact Methods for Load Impact Methods for Load Impact Methods for Load Impact (Protocols 11 (Protocols 11 (Protocols 11 (Protocols 11 (Protocols 11 – – – – – 16) 16) 16) 16) 16) Guidance Guidance Guidance Guidance Guidance Guidance Guidance Estimation for Long Term Estimation for Long Term Estimation for Long Term Estimation for Long Term Ex Ante Estimation Ex Ante Estimation Ex Ante Estimation Ex Ante Estimation Ex Ante Estimation Ex Ante Estimation Ex Ante Estimation Resource Planning Resource Planning Resource Planning Resource Planning Sampling Sampling Sampling Sampling Sampling Sampling Sampling (Protocols 17 (Protocols 17 (Protocols 17 (Protocols 17 (Protocols 17 (Protocols 17 (Protocols 17 – – – – – – – 23) 23) 23) 23) 23) 23) 23) Portfolio Portfolio Portfolio Portfolio Portfolio Portfolio Portfolio Portfolio Effects Portfolio Effects Analysis Analysis Analysis Analysis Analysis Analysis Analysis (Protocol 24) (Protocol 24) Sampling Sampling Sampling Sampling Sampling Sampling Sampling (Protocol 25) (Protocol 25) (Protocol 24) (Protocol 24) (Protocol 24) (Protocol 24) (Protocol 24) Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Determine Additional Determine Additional Determine Additional Determine Additional Determine Additional Determine Additional Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Additional Needs for Other Applications Other Applications Other Applications Other Applications Other Applications Other Applications Other Applications Other Applications Other Applications Other Applications Requirements Requirements Requirements Requirements Requirements Requirements Long Term Resource Long Term Resource Long Term Resource Long Term Resource Long Term Resource Long Term Resource Long Term Resource (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) (Protocol 2) Planning Planning Planning Planning Planning Planning Planning (Protocol 3) (Protocol 3) (Protocol 3) (Protocol 3) (Protocol 3) (Protocol 3) (Protocol 3) Revise Methodology and Revise Methodology and Revise Methodology and Revise Methodology and Revise Methodology and Revise Methodology and Data Needs and Finalize Data Needs and Finalize Data Needs and Finalize Data Needs and Finalize Data Needs and Finalize Data Needs and Finalize Evaluation Plan Evaluation Plan Evaluation Plan Evaluation Plan Evaluation Plan Evaluation Plan Implement Data Implement Data Implement Data Implement Data Implement Data Implement Data Process Protocol Process Protocol Collection and Analysis Collection and Analysis Collection and Analysis Collection and Analysis Collection and Analysis Collection and Analysis (Protocol 27) (Protocol 27) Produce Report Produce Report Produce Report Produce Report Produce Report Produce Report (Protocol 25) (Protocol 25) (Protocol 25) (Protocol 25) (Protocol 25) (Protocol 25) (Protocol 25) (Protocol 26) (Protocol 25) (Protocol 25) There are 27 protocols established by the CPUC & extensive guidance for evaluation planning

7. The protocols were used by CA IOUs in their applications for DR programs for 2009-2011 • PG&E will file consultant reports documenting ex ante estimates (based on ex post) for a variety of resources • PeakChoice, a new menu style program • Critical peak pricing (C&I) • Capacity bidding • Demand bidding • Base interruptible program (firm service level) • Aggregator managed portfolio • SCE filed testimony applying the protocols to its 2 largest DR programs • Interruptible tariff and program • Air conditioner cycling program • SDG&E applied the protocols to three programs • A/C cycling • Capacity bidding • Critical peak pricing

8. California’s IOUs will continue to apply the protocols each year • The IOUs must conduct ex post evaluations and update ex ante estimates by April of each year • The recently filed ex ante estimates were done under tremendous time pressure and, in some cases, with limited data on new or significantly evolving programs • Incremental requirements by the CPUC to develop impact estimates by customer type and rate created the need to produce thousands of tables by program, customer type, year, weather conditions, etc. • The state of the art will continue to evolve as CA focuses significant resources on impact estimation so that DR resources will be used and useful • Protocols for CAISO resources and for operational impact estimation are scheduled to be addressed next • Load impact for settlement is being addressed by each utility as part of program design

9. For more information, contact Dr. Stephen S. George Principal Consultant Freeman, Sullivan & Co. 415 948-2328 StephenGeorge@FSCGroup.com The following slides provide a high-level summary of California’s demand response load impact protocols

10. The first 3 California load impact protocols focus on evaluation planning • Protocol 1: A plan must be developed consistent with the requirements of protocols 2 and 3 and must include a budget and schedule • Protocol 2: The planmust delineate whether the evaluation will also address needs beyond the minimum requirements and, if so, describe what requirements will be met • Protocol 3: The plan must also delineate which of the issues depicted in the next slide are intended to be addressed

11. What is the Should Data Required Be Pooled Level of Ex Post Only Across Statistical or Ex Ante Utilities? Precision? As Well? Is An External Control Are Group Persistence Needed? Estimates Needed? Do You Need to Know the # Evaluation Evaluation Are Impacts of Free Plan Plan Needed for Riders or Geographic Structural Sub - regions? Benefiters ? Do You Need Are Sub - hourly to Know Why Impact The Impacts Estimates Are What Needed? They Are? What M&V & Are Impacts Survey Are Impacts Needed for Activities are Needed for Customer Required? Additional Segments? Day Types? Planning Protocol 3 requires delineating which of the following issues will be addressed What is the What is the Should Data Should Data Required Required Be Pooled Be Pooled Level of Level of Ex Post Only Ex Post Only Across Across Statistical Statistical or Ex Ante or Ex Ante Utilities? Utilities? Precision? Precision? As Well? As Well? Is An External Is An External Control Control Are Are Group Group Persistence Persistence Needed? Needed? Estimates Estimates Needed? Needed? Do You Need Do You Need to Know the # to Know the # Evaluation Evaluation Are Impacts Are Impacts of Free of Free Plan Plan Needed for Needed for Riders or Riders or Geographic Geographic Structural Structural Sub Sub - - regions? regions? Benefiters Benefiters ? ? Do You Need Do You Need Are Sub Are Sub - - hourly hourly to Know Why to Know Why Impact Impact The Impacts The Impacts Estimates Estimates Are What Are What Needed? Needed? They Are? They Are? What M&V & What M&V & Are Impacts Are Impacts Survey Survey Are Impacts Are Impacts Needed for Needed for Activities are Activities are Needed for Needed for Customer Customer Required? Required? Additional Additional Segments? Segments? Day Types? Day Types?

12. Protocols 4 through 10 concern ex post evaluation for event based resource options Ex Post Evaluation for Event Based Resources Event Based Pricing Direct Load Control Callable DR Protocols for Ex Post Evaluation of Event Based DRR Protocol 4: Impact estimates must be provided for each hour for each of the day types identified in Protocol 8 Protocol 5: The change in energy use for the year must also be estimated Protocol 6: Uncertainty adjusted impacts must be provided for at least the 10th, 50th and 90th percentiles Protocol 7: The impact estimates must be reported in specific tabular form delineated in this protocol for each day type Protocol 8: Impact estimates must be provided for each event day and for an average event day Protocol 9: Lists the statistical tests and measures that must be reported if day matching methods are used for impact estimation Protocol 10: Lists the statistical tests and measures that must be reported if regression methods are used for impact estimation Guidance and Recommendations for Ex Post Impact Evaluation of Event Based DRR

13. The template from Protocol 7 delineates the minimum output that is required for each day type

14. Guidance and Recommendations for Ex Post Impact Evaluation of Event Based DRR Additive Adjustment Scalar Adjustment Weather Based Adjustment The protocol document provides guidance and recommendations on a variety of methods and issues for ex post evaluation of event based DR options Day Matching Methods Regression Methods Other Methods Potential Bias Omitted Variables Wrong functional form Simultaneity Errors in Variables Influential data Select Test Days Sub-metering The Advantages of Repeated Measures Smaller sample sizes No external control Allows individual customer regressions More robust specification Select Reference Methodology Engineering Analysis Issues: Gaming Pre-cooling Other Adjustments Incorrect Standard Errors Serial Correlation Heteroscedasticity Irrelevant Variables Duty Cycle Analysis Experimentation Selection of best Reference Level method: See Reference Level Example Flexibility of Regression Analysis Measurement & Verification Why are impacts what they are? Geographic Specificity Multi Day Events Participant Characteristics Estimating Impacts for Hours Outside the Event Period Weather Effects Quantifying the Impact of Event Characteristics

15. Ex Post Evaluation for Non-Event Based Resources Non-Event Based Pricing Scheduled DR Permanent Load Shifting Protocols for Ex Post Evaluation of Event Based Demand Response Resources Protocol 11: Impact estimates must be provided for each hour for each of the day types identified in Protocol 15 Protocol 12: The change in energy use for each month and for the year must also be estimated Protocol 13: Uncertainty adjusted impacts must be provided for at least the 10th, 50th and 90th percentiles Protocol 14: The impact estimates must be reported in a specific tabular form delineated in this protocol for each day type specified in Protocol 15 Protocol 15: Impact estimates must be provided for an average weekday and for the monthly system peak day for each month in which the DRR option is in effect Protocol 16: Lists the statistical tests and measures that must be reported if regression methods are used for impact estimation Protocols 11 through 15 pertain to ex post evaluation of non-event based resource options

16. Guidance and Recommendations for Ex Post Impact Evaluation of Non-Event Based DRR Regression Methods Engineering Methods Day Matching Potentially Applicable to All Non-Event Based Options Applicable Primarily to Permanent Load Shifting Options May be Applicable to Scheduled DR Demand Modeling Potentially Applicable to Pricing Options External Control Group May be Needed Options to Control for Selection Bias Guidance for non-event based DR options focuses on the important role of control groups Population Matching Incorporate Key Drivers in Regression Model 2 Stage Model: Estimate Probability of Participation Create Control Group From Existing or Future Participants

17. Ex Ante Estimation Event Based Resources Non-Event Based Resources Protocols for Ex Ante Estimation for All Demand Response Resources Protocol 17: Base ex ante estimates on ex post evaluations (including analysis of resources from other utilities if necessary) Protocol 18: Impact estimates must be provided for each hour for each of the day types identified in Protocol 22 Protocol 19: The change in energy use for each month and for the year must also be estimated Protocol 20: Uncertainty adjusted impacts must be provided for at least the 10th, 50th and 90th percentiles Protocol 21: The impact estimates must be reported in a specific tabular form delineated in this protocol for each day type specified in Protocol 22 Protocol 22: Impact estimates must be provided for a typical event day for event based resources and for an average weekday and the monthly system peak day for non-event based resources for each month in which the resource is in effect Protocol 23: Lists the statistical tests and measures that must be reported if regression methods are used for impact estimation Protocols 17 – 23 apply to ex ante estimation, which must be based on ex post evaluation

18. Guidance and Recommendations for Ex Ante Impact Estimation Ex Ante Scenarios Estimation Methods Impact Persistence Uncertainty in Key Drivers Conditions Within The Range of Historical Experience Incorporate Change Variables in Regression Model Conditions Outside The Range of Historical Experience Incorporate Change Variables in Regression Model and Test for Non-linearities Changes in Observable Participant Characteristics Experimentation Borrow Models and/or Results from Other Utilities Changes in Unobservable Participant Characteristics New Resource Options Methods for ex ante estimation will vary across ex ante scenarios

19. Protocol 24 addresses issues associated with impact estimation when multiple programs are called at the same time • DR resource evaluation must identify the correlations, synergies and overlaps across a set of DR resources • A judgmental determination of the impact of the magnitude of adjustment in program impacts should be made for all programs

20. The protocoldocument also contains guidance for estimating impacts for DR portfolios

21. Sampling is an important aspect of most impact evaluations

22. Sampling protocol 25 is designed to minimize bias in any samples that are used while not being overly prescriptive • The population(s) under study must be clearly identified and described • The sample frame(s) used to sample the population(s) under study must be carefully and accurately described along with any measures used to correct for differences between population and sample frame • A digital snapshot of the population and initial sample from the sample frame must be preserved • The sample design must be described in detail, including the distributions of population and sample points across sampling strata (if any) • The “fate” of all sampled observations must be tracked and documented throughout the data collection process • If significant sample attrition is found to exist at any stage of the research process (i.e., recruitment, installation, operation), a study of its impact must be undertaken • If selection bias is suspected, the evaluator must describe it as well as any efforts made to control for it

23. The reporting protocol (26) sets minimum requirements for documenting the analysis and reporting the results • Evaluation reporting has a variety of objectives • Describing the evaluation objectives and plan • Presenting the detailed impact estimates developed as part of the evaluation • Comparing these findings with resource goals and the impacts that have been used to report progress toward goals, and explain any differences • Thoroughly documenting the methodologies used in sufficient detail so that, given access to the same data and information, a trained evaluator would be able to reproduce the impact estimates that are reported • Reporting any deviations from the requirements of these protocols and the reasons why it was not possible to meet them • Providing recommendations regarding resource modifications and modifications to the impact estimates used for resource progress reports • Providing recommendations concerning future evaluation activities.

24. The final protocol (27) provides for public review and comment • A review and comment process will be used at three stages in the implementation of the load impact estimation effort • The evaluation plan used to develop the research questions to be answered and the corresponding methods to be used to answer them • The interim and draft final reports for all load impact studies conducted for demand response resources • Review of final reports to determine how comments were addressed • The Demand Response Measurement Evaluation Committee (DRMEC) will be used to initiate evaluation planning, review the final evaluation plan and review draft reports