DALRP Static Baseline Analysis

1. DALRP Static Baseline Analysis Miriam Goldberg Presented to the NEPOOL Markets Committee Meeting March 31, 2008

2. Background • On January 23, 2008, ISO-NE presented several issues with the Day-Ahead Load Response Program to the NEPOOL Markets Committee. • At that time, ISO-NE proposed indexing the DALRP Minimum Offer Price to the Forward Reserve Fuel Price Index using an implicit heat rate of 12.92 MMBtu/MWh. • The NEPOOL Markets Committee voted to implement the change effective February 8, 2008. • ISO-NE committed to a 60-day stakeholder process to evaluate alternative proposals.

3. Background (Continued) • Three Market Participants submitted alternative proposals to ISO-NE in February 2008. • ISO-NE retained KEMA to conduct an independent analysis of the key alternative proposals.

4. KEMA Experience and Qualification • California Energy Commission: Protocol Development for Demand Response Calculation – Findings and Recommendations • ISO-NE: Demand Response Reserves Pilot Technical Support • NYSERDA: Technical Support for Demand-Responsive Load Pilot • Gruenich Associates: Technical Support for California Demand-Response Program • EPRI: Methods Assessment for Load Profiling for Retail Market Settlement

5. Alternative Proposals Evaluated • IECG Proposal – Option 1: • Lower the effective heat rate below the present 12.92 MMBtu/MWh level. • IECG did not object to EnerNOC alternative proposal described below. • EnerNOC Proposal: • DALRP Minimum Offer Price restored to $50/MWh. • Implement a New Customer Baseline methodology for all ISO Load Response Programs. • Under EnerNOC’s proposal, the Customer Baseline is calculated using a minimum of 15 programs days over a trailing 45 program-day window using a weighted-average methodology.

6. Analysis Methodology

7. Analysis Objective • Determine how accurate each Customer Baseline methodology is at predicting a customer’s actual hourly load with no load reduction event. • Does the baseline tend to be high or low, and by how much? • How variable is the difference between the estimate and the actual load?

8. Analysis Approach • Data analyzed • Actual 2007 hourly loads from a sample of customers (assets) that DID NOT participate in the DALRP or Real-Time Price Response Program. • Hourly Customer Baseline calculations for each asset • ISO-NE’s current methodology. • Each Alternative Proposal. • “Weather adjustment” to the calculated Customer Baseline consistent with ISO-NE’s current methodology.

9. Measuring Baseline Error • In each DALRP cleared hour, for each asset, calculate the difference between the asset’s adjusted Customer Baseline and its actual load. • Looking only at NONPARTICIPATING assets. • Difference between baseline and actual is baseline error for those assets and hours. • Summaries of baseline errors over customers and hours measure overall baseline method accuracy.

10. Data Sources • ISO-NE provided hourly data from 533 non-participant commercial and industrial customers (assets). • The data used in the analysis included all assets that had a full year of data and no missing hours.

11. Determining Cleared Hours for the Analysis • Assets are assumed to bid at minimum offer price every eligible day after an assumed starting date. • Evaluated different starting dates. • Offer clears when minimum offer price is less than or equal to Day Ahead LMP. • Based on actual 2007 Day Ahead LMPs in the Load Zone where each asset is located. • Each event started with the first cleared hour after 7am, and ended at 6pm.

12. Baseline Error • Error = (Baseline) – (Actual Load) • Positive error means baseline is over-estimating actual load. • Therefore potentially overstating load reduction.

13. Median Relative Error • Measures bias or systematic error • Error = (Baseline) – (Actual Load) • Monthly, asset-level relative error = (mean asset error)/(mean actual Load) • Allows comparison across assets of varying sizes • Median relative error = Median of monthly relative errors over assets • by month or day

14. Asset Specific Hourly ErrorError= (Baseline) – (Actual Load)

15. Asset Specific Relative ErrorRelative Error = Average Error/Average Load Monthly Relative Error uses averages over all cleared hours in a month.

16. Median Monthly Relative Error • Calculated 533 asset-level relative errors for each month. • Identified the Median Relative Error in each month. • Equal number of assets have relative errors above and below the median. • Because error is in percentage terms, all assets are equally weighted. • Median is less affected by outliers than Mean.

17. Starting Date Matters • Market Participants can begin offering an asset into the DALRP on any non-holiday weekday (after baseline is established). • KEMA’s analysis shows that the Relative Error of all the Customer Baselines is dependent on the date which an asset starts offering in the DALRP. • KEMA evaluated 4 different offer start dates: • January 2007 • April 2007 • August 2007 • September 2007

18. First Offer Date: January 2007

19. First Offer Date: April 2007

20. First Offer Date: August 2007

21. First Offer Date: September 2007

22. Comparing Alternative ProposalsMedian Monthly Relative Error, DALRP Offers start August 1, 2007

23. Relative Root-Mean-Square Error (RMSE) • Measures typical error magnitude • Combines systematic bias and variability • Relative RMSE is always positive • Greater relative RMSE indicates greater bias and noise (variability) in the error estimate. • RMS = Square root of average squared value • square each value, average the squares, take the square root of the average • Relative RMSE= RMS Error / RMS Load

24. Comparing Alternative ProposalsMedian Relative RMSE, DALRP Offers start August 1, 2007

25. Number of Cleared Days • Proposals have different Minimum Offer Prices. • Minimum Offer Price compared to actual 2007 Day Ahead LMPs. • At $50/MWh, all eligible days clear. • As Minimum Offer Price increases, fewer eligible days clear.

26. Number of Cleared Eligible Days continued The number of cleared days in this table is based on 2007 Connecticut Day Ahead LMPs.

27. Monthly Total MWh Error • The sum of all cleared hourly errors (Baseline – Actual Load) for the month across all assets. • Negative errors (hours when Actual Load is greater than the Baseline) are included in the sum. • Unlike median relative error, assets are not all weighted equally.

28. Comparing Alternative ProposalsMonthly total MWh Δ (CBL-MW), DALRP Offers start August 1, 2007

29. Monthly Sum of Payments • All cleared hourly errors are monetized at the hourly Day-Ahead LMP. • Hourly payments are summed across all hours in the month and across all assets.

30. Comparing Alternative ProposalsMonthly Sum of Payments (+/-), DALRP Offers start August 1, 2007

31. Lessons Learned • Various measures of error tell similar stories for this study. • Relative RMSE results mirror relative error results. • That is, most of the baseline error is systematic, not noise. • Monthly total error results are driven by combination of error magnitude and number of cleared days. • Same general trends found as for relative measures. • Indicates similar patterns across asset sizes.

32. Conclusions on Baseline Options • Original DALRP design ($50/MWh Minimum Offer Price) has highest overall error, by all measures of error. • Equivalent to very low heat rate or clearing every day. • Current DALRP design (Minimum Offer Price indexed to 12.92 MMBtu/MWh) has lowest overall error. • Minimum Offer Price indexed to 10.85 MMBtu/MWh (lowest examined) distinctly worse than others. • Other heat rate options evaluated fall in between.

33. Conclusions--continued • EnerNOC option • Up to two months after starting point errors are similar to original DALRP design. • By 4th month after starting point errors are similar to current DALRP design.

34. Thank you for your attention.