SEEM 94 Calibration to Single Family RBSA Data Analysis and proposed actions

1. SEEM 94 Calibration to Single Family RBSA DataAnalysis and proposed actions RTF SEEM Calibration Subcommittee May 7, 2013

2. Goals for today’s Subcommittee Meeting • Review the following presentation in detail • Consensus on next steps • Are there any needed changes to the analysis? • Is there subcommittee consensus that the RTF should make a decision stating SEEM94 is calibrated? • Receive suggestions from the subcommittee for improvements in the presentation • Does it adequately tell the full story? • Is it the appropriate tool present to the RTF (assuming previously covered sections will be skimmed over)?

3. SEEM 94 Calibration to Single Family RBSA DataAnalysis and proposed actions Regional Technical Forum May 21, 2013

4. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

5. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

6. Purpose of Calibration

7. Align SEEM with Measured Energy Use • The SEEM model is used to estimate energy savings for most space-heating-affected residential UES measures using the “calibrated engineering” estimation procedure (see section 2.3.3 of guidelines) • Heat Pumps and Central AC (ASHP, GSHP, DHP) • Weatherization • New Homes • Duct Sealing • Space Conditioning Interaction Factor • Goal: Ensure SEEM94’s results are grounded in measured space heating energy use of single family homes. Use RBSA as source of measured data. Background - Purpose

8. RTF Savings Guidelines Background - Purpose

9. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

10. RTF Decision History Background - History

11. RTF Decision History (Continued) For “model is calibrated” decisions… Calibration Methodology: Use available house and operation characteristics data from billing/metering studies to develop inputs to SEEM runs; Adjust SEEM thermostat setting input to achieve a good match (on average) between SEEM output (annual heating energy use) and billing/metering study results. Note: The data sources used were free of (or mostly free of) supplemental fuel usage (wood, propane, oil, etc.) Collection of reliable electric and gas usage data for space heat consumption is relatively easy compared to other fuels. Background - History

12. SF Calibration to RBSA - Recent History

13. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

14. Methodology – Overview

15. Two Sources of Heating Energy Estimates RBSA-PRISM.Estimates of annual “space heating use” for each house determine by using PRISM • PRISM is a “change-point” regression model that uses billing data to estimate temperature-sensitive use • PRISM analysis based on monthly billing data (at least 2years) SEEM.Estimated annual space heating energy use for each house based on SEEM engineering model • RBSA individual home characteristics (e.g., thermal envelop, heating system type, duct tightness) used as model inputs; • Initial model runs use thermostat set to 68°F day & night • SEEM is a one-zone model, so t-stat setting input represents the average setting for the entire house • Actual t-stat settings are not well documented (occupant reported settings are unreliable, especially for “zonal systems”) • Thermostat setting will be used (step 2, below) as the “calibration knob” Methodology - Overview

16. Step 1 (Regression) Use regression techniques to identify building characteristics that drive systematic differences between SEEM(68°F) and PRISM space heating energy use estimates. Methodology - Overview

17. Step 2 (Calibration) Use regression results to determine thermostat set-point that will align (i.e., “calibrate”) SEEM with PRISM annual space heating use • Calibration based on comparing average of all SEEM (68) annual estimates to average of all PRISM annual estimates • Calibration is based on building characteristics identified in regression. • SEEM run for each house at varying “day-time” thermostat settings, with “night-time” thermostat settings based on occupant reported setbacks Methodology - Overview

18. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

19. Data Sources • * Using a pre-release version of the database for this analysis . Data Source used in this calibration: Underlying database* for the Single Family Residential Building Stock Assessment (2012) • RBSA study’s database offers recent billing analyses results and detailed house characteristics on 1404 single family houses in the Region. • RBSA data allows inputs for SEEM runs to be well defined for individual homes. Methodology - Data

20. Methodology - Data

21. Detail: Non-Lighting Internal Gains • *Hendron, Robert. "Building America Research Benchmark Definition, Updated December 20, 2007." NREL/USDOE EERE. January 2008. NREL/TP-550-42662 • Equation: • Based loosely on Building America Benchmark* • Used the original equation and values (averaged) to determine average internal gains for RBSA homes. • Original equation also includes Number of Bedroom and Finished Floor Area terms • Set Number of Bedrooms and Finished Floor Area terms to zero and adjusted Number of People term to achieve same average internal gains for RBSA homes. • Building America Benchmark based on • “The appliance loads were derived by NREL from EnergyGuide labels, a Navigant analysis of typical models available on the market that meet current NAECA appliance standards, and several other studies. ” • “The general relationship between appliance loads, number of bedrooms, and house size, was derived empirically from the 2001 RECS. ” Methodology - Data

22. Realistic SEEM Simulations Not Feasible/Possible for All Homes in RBSA • Resulting House Count: 1011 • These issues overlap on some houses, so the sum of the counts cannot be subtracted from 1404 to get 1011. Methodology - Data

23. Data Filters Excluded Some RBSA Homes • Gas Billing converted to kWh/year using reported AFUE • Resulting House Count: 293 • (The counts for each item overlap here, too) Methodology - Data

24. Additional Data Filter for PRISM Excluded Additional Homes Exclude any home that had an out-of-range PRISM T-balance for one or more components. • The PRISM analysis restricted balance point temperatures to be between 48 and 70 ⁰F. • T-balance below 48⁰ is plausible. • T-balance above 70⁰ is not physically plausible. We filter these out since such values are evidence of a poor PRISM fit. • Our 293 sites’ T-bal values include… • 10 that defaulted to 70⁰when PRISM’s initial fit exceeded the max. (Excluded from analysis.) • 16 that defaulted to 48⁰when PRISM’s initial fit was below the minimum. (Kept) • 267 whose PRISM balance points were within the acceptable range. (Kept) • This leaves 283 sites for the present analysis. Methodology - Data

25. Final Data SetSEEM values calculated with t-stat = 68°F (constant) Seem (68) Heating Energy (kWh) Methodology - Data

26. Overview • Background • Purpose • History • Methodology • Data • Regression • Calibration • Discussion • Proposal

27. Methodology - Regression

28. Regression Overview (1) • Analysis Identify and quantify any systematic patterns (trends) in the differences between SEEM(68°F) and PRISM savings estimates (∆ kWh = SEEM(68°F) kWh ‒ PRISM kWh. • Systematic means “explained by known variables.” (Example: SEEM(68°F) kWh tends to exceed PRISM kWh in cooler climates.) • Tacit assumption: PRISM estimates roughly unbiased. • Definitions • “PRISM kWH” = Heating energy use via billing analysis; from RBSA SF dataset. • “SEEM(68°F) kWh” = Heating energy use via SEEM runs using house-specific characteristics data from the RBSA SF dataset with thermostat set to 68°F Methodology – Regression

29. Regression Overview (2) • Problem is multivariate… • A single underlying trend (example: ∆ increasing with heating use) may appear in multiple guises (∆ increasing with HDD, or with U-value, or with building heat loss) • Approach is multiple regression… • Compare PRISM kWh with SEEMkWh when SEEM is run with a constant T-stat setting (68°F day, 68°F night.) • Y-variable is the percent difference between SEEM kWh and PRISM kWh (when SEEM uses T-Stat=68°F). • X-variables are physical characteristics known through RBSA. (Specifying the x-variables is a large part of the work.) Methodology – Regression

30. Setting up the Regression (1) Primary interest is in differences between SEEM(68) kWh and PRISM kWh—the Y-variable must capture these differences. • Heteroskedasticity. The SEEM(68) /PRISM differences generally increase in magnitude in proportion to SEEM(68) kWh (or PRISM kWh). (See earlier graph.) • Measurement error (random noise). As estimates of heating kWh, SEEM(68) and PRISM both have substantial standard errors. Methodology – Regression

31. Setting up the Regression (2) • Note choice of signs: means SEEM > PRISM. • What goes in the denominator? “??” = “Actual kWh” would be ideal. • Using SEEM kWh or PRISM kWh would skew y-values. (Next slide.) • Log-transforms (closely related) not quite right either. • Instead, divide by midpoint: “??” = (SEEM + PRISM)/2. (Two slides down.) Methodology – Regression

32. Dividing by PRISM kWh magnifies differences where PRISM’s random error happens to be negative (since these values get artificially small denominators). This biases the percent differences upwards. Methodology – Regression

33. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. Methodology – Regression

34. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. Methodology – Regression

35. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. Methodology – Regression

36. Upward bias (mostly) goes away when we divide by the value halfway between SEEM(68°F) and PRISM. Methodology – Regression

37. Building the Regression Model • Goal is to identify variables that lead to systematic differences between SEEM(68°F) and PRISM. • “Lead to” is only seen in rough trends (think: correlation). • Looking to capture unknown effects – nota physical model. • Model development is iterative. • A variable may be weakly correlated with raw y-values but strongly correlated with y’s that have been adjusted to account for some other variable’s influence. Methodology – Regression

38. Important Limitations • Avoiding Colinearity - When a potential x-variable closely tracks some combination of variables that are already included. • Example – Including both heat loss rate and vintage • This redundancy leads to unstable model fits. • Threshold for “tracks too closely” gets low when the usual suspects are around: High noise / faint signal / small sample. • Pursuing Parsimony. General principle: Don’t over-fit the data (by including too many explanatory variables). • Incomplete data variables. Some variables (e.g., duct tightness and infiltration) aren’t known for very many houses. Methodology – Regression

39. Prominent x-variable candidates • Characteristics that likely influence differences between SEEM(68°F) and PRISM estimates of use • Thermal efficiency drivers (U-values, duct tightness, infiltration, …) • Heating system type • Climate (i.e., HDDs) • Following graphs illustrate “influence” of several variables (separately) on percent difference between SEEM(68°F) and PRISM Methodology – Regression

40. Methodology – Regression

41. Methodology – Regression

42. Methodology – Regression

43. Insulation Variables The big surfaces: Wall, ceiling, and floor. • Express in terms of heat loss (U-values, weighted by surface area as appropriate) • We separate out Floor U because of different foundation types. • One variable accounts for ceiling and wall heat loss. • Another variable accounts for floor heat loss in crawlspace homes. Methodology – Regression

44. Variable for Wall/Ceiling U Applies to all homes (regardless of foundation type). A simple indicator variable: “Wall/Ceiling Insulation is Poor” if Wall u-value > 0.25, OR Ceiling u-value > 0.25, OR Both u-values > 0.25. This variable captures the main effect of the weighted average. (See next slide) Methodology – Regression

45. Methodology – Regression

46. Variable for Floor U Particularly interested in crawlspace heat loss since crawlspace insulation is a common measure. Variable definition: “Yes/No” indicator for uninsulated crawlspace. Note: Sites with basements, slabs, and insulated crawlspaces all have Uninsulated Crawl = “No” Methodology – Regression

47. Do these indicator variables really capture the insulation effects? The next two slides compare various u-values’ relationships with • Unadjusted (raw) percent differences; • Percent differences that have been adjusted for the two insulation variables included in the regression. Methodology – Regression

48. Methodology – Regression

49. Methodology – Regression

50. Heating System Variable Four distinct heating systems in the sample: Electric zonal Electric FAF Gas FAF Heat pump After controlling for insulation, heating system effect appears to be captured with just two groups: “Electric Resistance” = Electric zonal / Electric FAF “Gas/HP” = Gas FAF /Heat Pump Parsimony: two is better than four! Methodology – Regression