Monitoring for Technical Performance, Accountability, and Consistency

1. Monitoring for Technical Performance, Accountability, and Consistency U.S. Department of Energy Project Management Center February 2010 Presenter: Kelly Cutchin, SMS Bethesda, MD. / February 23, 2010 NASCSP Mid-Winter Training

2. Regulatory Monitoring Requirements • The PMC “shall monitor and evaluate [Grantees] . . . through on-site inspections, or through other means”. • The Secretary of Energy, DOE Inspector General, [and others] shall have access to any books, documents, papers, information, and records of any weatherization project… for the purpose of audit and examination.

3. Regulatory Monitoring Requirements • The Grantee must conduct comprehensive monitoring of each subgrantee at least once a year. • The comprehensive monitoring must include review of client files and subgrantees records, as well as actual inspection of at least 5 percent of the DOE-funded completed units of each subgrantee.

4. Monitoring Requirements Specified in Guidance • Grantees are monitored by PMC Project Officers against current, approved annual plans. • PMC Project Officers periodically accompany Grantees to observe their monitoring and oversight of subgrantees. • Written notice of planned visit by PMC Project Officer provided 30 days in advance. • Written monitoring report due to Grantee within 45 days after site visit.

5. Monitoring Requirements Specified in Guidance • Monitoring report provided to Grantee should include specific items addressed and due date for follow-up actions. • In a year with no site visit, desk monitoring (i.e. WinSAGA, FSR’s PSR’s, etc.) of Grantee by PMC Project Officer is required.

6. Monitoring Requirements Specified in Guidance • PMC systematic process for programmatic review of Grantee: • Determine if new regulatory and legislative changes are implemented; • Follow through on noted action items; • Review Grantee’s Policy & Procedures Manual and/or identify procedural issues through dialogue • Review client files; • Review monitoring files; • Review contract files; • Review fiscal records; • Review reported items/issues.

7. Monitoring Requirements Specified in Guidance • PMC systematic process for programmatic review of Grantee (cont.): • Review T&TA activities; • Review leveraging activities; • Ensure effective monitoring of all subgrantees; • Identify Grantee WAP successes.

8. Monitoring Requirements Specified in Guidance • PMC fiscal review of Grantee: • Perform overall fiscal due diligence; • Ensure costs being charged are proper; • Compare costs in relation to both per unit and overall production; • Verify high cost items are getting DOE pre-approval; • Review A-133 audit.

9. Federal Monitoring Reports • Reports should include the following: • Observations about areas in the WAP that the Grantee is doing well in terms of managing the Program according to DOE and state regulations, as well as being creative in program operations. • Recommendations on how the Grantee can improve their Program, along with consideration of additional T&TA initiatives. • Violations of Program rules and regulations must be documented along with remediation suggestions and a definitive timeline for corrective actions.

10. Subgrantee Monitoring Reports • Reports should include the following: • Observations about how the subgrantee is achieving their assigned goals and spending funds appropriated, as well as undertaking any creative programs. • Recommendations on how the subgrantee can improve program and technical field performance, along with consideration of appropriate T&TA activities. • Violations of Program rules and regulations must be documented along with remediation suggestions and a definitive timeline for corrective actions.

11. Local Agency Inspection Reports • Reports should include the following: • Detailed confirmation that the weatherization work completed in a dwelling was what was recommended via the audit or priority list. • Verification that everything billed was actually installed in the dwelling. • Identification of poor workmanship and tracking record of “call-backs” and re-inspections by the local agency Inspector.

12. Monitoring Requirements • Regulations and policy mostly address programmatic and fiscal monitoring, which targets: • Compliance with regulations and policy; • Prevention of waste, fraud, and abuse.

13. The Need for Technical Monitoring • Regulatory and policy monitoring largely do not assess how well a state or local agency achieves its core mission, which is to: • Increase the energy efficiency of dwellings owned or occupied by low-income persons; • Reduce client’s total residential expenditures; • Improve client health and safety; • Achieve a stated goal for DOE completed weatherized units.

14. The Need for Technical Monitoring • Technical monitoring includes verifying what was done in a dwelling was not only done right, but was the right thing to do. • Sometimes a Monitor needs to get his/her head into the nitty-gritty of a house to ensure quality work was completed!! ..

15. What Is Technical Monitoring? • What weatherization services are provided to eligible low-income households? • Are the appropriate services being provided? • Are measures installed properly? • Could service delivery be improved to: • Increase savings? • Lower costs? • Raise production and reduce call-backs? • Better ensure client health and safety?

16. What Is Technical Monitoring? • Technical monitoring assesses: • Energy Audit and/or Priority List procedures; • Diagnostic testing • Measure selection • Health and safety protocols; • Installation standards; • Client education; • Quality control efforts; • Evaluation/feedback mechanisms.

17. What Is Technical Monitoring? • Post-weatherization homes should not look like this home. • Snow indicates slope section where there is insulation. • No snow indicates slope section where there is no insulation (warm roof has melted the snow).

18. Energy Audit Procedures • Is the energy audit conducted with sufficient technical rigor? • Does input data represent house being audited? • Are local labor and materials costs used? • Are actual fuel costs used? • Are assumptions reasonable? • Are certain measures being forced or eliminated?

19. Energy Audit Diagnostics • What diagnostic testing is used to guide measure selection? • Blower door: Are results actually used? • Is a pre and post blower door conducted? • Zone pressures: To locate big bypasses and verify effectiveness of air sealing; • Pressure pan testing of ducts located outside conditioned space;

20. Energy Audit Diagnostics • Furnace efficiency testing; • Worst case combustion appliance zone (CAZ) spillage test; • Minimum ventilation testing via ASHRAE standards.

21. Measure Selection • Is the energy audit actually used to determine what weatherization services are most appropriate for a particular house? • Or, is the energy audit conducted just because it’s required to be in the file? • Does the Auditor know when to use an audit instead of a Priority List?

22. Measure Selection • Are energy audit recommendations sound and reasonable? • Are energy audit recommendations implemented? • If so, are they implemented based on best SIR or cost per measure? • Is a priority list being followed? • Are measures skipped? • Are measures installed that are not recommended by energy audit or priority list?

23. What Measures Typically Save the Most Energy? • Air sealing (depending on existing leakage); • Duct sealing/insulation (depending on where ducts are located); • Attic and wall insulation; • High-efficiency heating and cooling equipment; • Refrigerator replacement.

24. Other Measures with High SIR’s • Water heater tank & pipe insulation; • Low-flow showerheads & faucet aerators; • Incandescent-to-CFL lighting conversions; • Programmable thermostats (in theory); • Weatherstripping.

25. Health & Safety Protocols • Is a comprehensive moisture assessment being performed? • Visual inspection for mold and mildew related problems. • Is home tested for health and safety before and after weatherization? • Inspect for building structure problems; • Inspect for electrical problems; • Inspect for combustion related problems.

26. Health & Safety Protocols • Are weatherization workers practicing Lead-Safe Weatherization (LSW) practices? • Are work crews/contractors properly trained and carrying the proper safety equipment (i.e. respirators, Tyvek suits, safety glasses, first aid kit, etc.) to job sites? • Is equipment used? • Is equipment functioning properly?

27. Health & Safety / EPA Certified Renovator • Weatherization workers disturbing 6 or more square feet of lead-based paint in a room or where 20 square feet or more of lead-based paint is disturbed on the exterior of the dwelling must attain Certified Renovator (CR) accreditation. • Monitors should ensure that appropriate Auditors, Monitors, crew members, and contractors have attained CR status.

28. Installation Standards • Are measures installed such that energy savings persist over the life of the measures? • Proper materials • Quality products while cost effective? • Quality workmanship • Proper training for installers? • Verified through inspections and monitoring? • Is the dwelling likely to exist long enough to recoup the investment in weatherization? • If not, this would constitute a “walk-away”.

29. Education • Is the client informed about the weatherization measures installed in their home? • Are measure-specific maintenance issues discussed? • Are other energy-saving practices reviewed with the client?

30. Quality Control • What are final inspection procedures? • Is the auditing and inspection processes separate and independent from installations? • Is there a written inspection form or tool? • Are there written technical program standards? • Do local agency staff know standards document exists? • Are standards periodically updated?

31. Evaluation Mechanisms • Are pre- and post-weatherization utility bills reviewed to determine actual energy savings? • Is a Grantee program evaluation periodically conducted? • How do local agencies evaluate their program performance?

32. Important, But Difficult • Technical monitoring is particularly important in light of significant ARRA funding and impending national evaluation; • Technical proficiency is required to provide effective technical monitoring; • Constrained Grantee budgets force staff with non-technical backgrounds to monitor an increasingly technical program; • Grantee monitoring is a valuable training opportunity.

33. Monitoring Initiatives • Technical monitoring checklist has been developed. • Only a start. • Creates standard process across all Grantees. • Weatherization Plus Committee working on “Consistent Delivery of Quality Services” • Encompasses technical training and monitoring; • Training needs assessment; • Define job descriptions, core competencies, and skill sets for various weatherization positions, including monitors.

34. Pipe Chase Air Sealing • Air sealing pipe chases like this prohibits air movement between unconditioned (non-living) and conditioned (living) spaces. • Monitoring should inspect quality and completeness of the work.

35. Infrared Thermography • Infrared camera technology allows the user to visually see heat loss in a dwelling. • Infrared cameras can’t see through anything. They read surface temperature only. • A trained technician can interpret the visual heat picture to determine the lack of or effectiveness of installed sub-surface insulation. • Infrared cameras serve as a great monitoring tool to verify proper insulation installation. An excellent tool for Monitors at all levels and even for insulation installation technicians.

36. Barriers to Consistency • John Krigger’s list of barriers: • Opting out of measures; • Lack of good equipment; • Excessive job documentation; • Excessive and disorganized guidance; • Lack of technical program standards; • Lack of qualified technicians; • Lack of consistent, focused client education; • Lack of technical plan for domestic water heating.

37. Accountability • Leadership for technical performance starts at the local level. • Grantees provide accountability through T&TA and monitoring. • The PMC oversees Grantee monitoring activities. • Program regulations and guidance outline Grantee monitoring responsibilities.

38. Impact of ARRA Funding • With increased funding comes increased production and thus increased monitoring. • Monitoring at all levels becomes the key to maintaining consistent workmanship and identifying inferior workmanship. • Monitoring of work performed by new hires is key to know if existing training is sufficient and to plan for future training. • With large funding increase, monitoring will be critical in identifying any issues of waste, fraud, and abuse in the program.

39. Next Steps? • Publicize the skills, training, and minimum requirements expected for technical monitoring? • Revisit and revise monitoring requirements in Program Guidance? • Develop training and peer exchange opportunities for new hires?

40. Next Steps • Require certification? • Of Auditors, Crew Chiefs? • Of crews, contractors, and installers? • Of Monitors, Inspectors? • Develop a uniform certification standard for various job titles throughout the Program?