Net-to-Gross: A Few Observations Vis- á -Vis the Long Term

1. Net-to-Gross: A Few Observations Vis-á-Vis the Long Term CALMAC San Francisco, California July 17, 2007 Michael RufoItron Inc. 1111 Broadway, Suite 1800 Oakland, CA 94607 510-844-2881

2. Theory • Love ‘em, hate ‘em, but… • Free ridership and spillover/MT • Important concepts, reasonably well understood, important to program design, forecasting/procurement, policy making • But terms and measurements don’t always capture and convey what we want • Free ridership and spillover/MT • Two sides of the same coin • It’s short-sighted to focus on one while ignoring the other • What matters is long term market change: • Totality of: Direct program participation, market effects, codes/standards, and other influences • Free ridership is limited as a short-term, participant-driven metric

3. Why Worry About Free Ridership? • Non-participant’s perspective • Don’t give my money to someone else to do something they were doing anyway • Program efficacy • Competition for scarce public purpose dollars • Concerns can’t be cavalierly dismissed • and someone will always raise them! • But… • when programs are effective over the long-term • and participation is widespread • concerns can be mitigated, if not eliminated

4. Program Effects are Often Acceleration (RER 2001)

5. Practice • But…there are problems, mountains of them • Adoption is non-linear and so is free ridership • Markets are dynamic and when interventions succeed change is accelerated • Line AD in previous slide is not what we observe • What is a true naturally occurring baseline 25 years after the first market interventions? • Today’s free riders are often yesterday’s market effects • Measurement techniques are limited • No pure control groups • Whatever happened to experimental design! • The vagaries of self reports • The pain of market tracking

6. Practice • Measurement challenges partially related to under and inconsistent investment • ~$1-2B/yr over past 15 years on programs • How much on evaluation and longitudinal baselines? • Evaluation efforts spotty and often half hearted • Real-learning the same inconclusive stuff over and over instead of conducting rigorous, consistent, and, yes, sometimes more costly, long-term research • No surprise we have very few reliable longitudinal data sets of market saturation, penetration, costs • Nationally, reported program data is weak! • Doesn’t support econometric analysis

7. NTG Application to Implementers • Extreme cases • Linear scaling of reward/penalty, threshold triggers • No financial feedback • Nationally, some fallout from the1990s? • Fear and stipulation • Let’s call the whole thing off • Maybe if we don’t measure it it will go away • But ex post NTG can provide vital feedback • Critical to improving program design • In some cases, partially reward/penalize depending on what administrators can realistically control • Not a substitute for multi-year market effects analyses

8. NTG Under Aggressive Program Funding • Over the long term • EE program $ essentially savings and investment fund • Customers are really using their own money • To enhance buying power • Stimulate new markets • Mitigate market barriers • Explicit in “Use it or loss it” approach for large C&I • Short-term free ridership becomes less of a concern • If long-term participation is widespread • Significant market effects are accomplished • Programs/policies adapt quickly to accomplishments/failures • Latter requires continuous measurement of both short-term program impacts and long-term market effects

9. Conclusions • Not measuring should not be an option • But traditional focus on “free riders” is suboptimal and can lead to wrong conclusions • “Free rider” term itself problematic/derogatory • Not consistent with traditional economic use of term • What matters are • Short-term “marginal program effectiveness (MPE)” • Let’s find a better term that gets at what we care about • Long-term market effects • And, yes, associated program attribution.

10. Conclusions • Attribution is obviously challenging • Precise attribution will always be difficult • “Chunky” attribution less so • Results should be used to • Optimize program and portfolio design • Know when to • declare victory, admit failure, redesign, or move on • Appropriately direct and motivate implementers, w/out • Penalizing for factors outside direct control • Creating perverse incentives • (e.g., maximizing short-term versus long-term impacts) • Improve forecasts and influence on procurement

11. Wrap We should not let short-term, and sometimes controversial applications of NTG, overly influence research and evaluation planning objectives; which should be driven by both short- and long-term perspectives on measuring changes in total societal energy efficiency (and energy consumption) in ways that are robust in the face of changing program strategies and policy regimes…Nor should we let measurement challenges lead to measurement avoidance.