Module 1: Why Use Statistics? An Introduction to the ERP Case Study

1. Module 1: Why Use Statistics? An Introduction to the ERP Case Study Steve DeGabriele (Massachusetts DEP) For the Workshop: Using Essential Statistics for Effective Program Management Presented at:2008 Symposium on Innovating for Sustainable Results Chapel Hill, North Carolina January 7, 2008

2. Overview: Using Essential Statistics forEffective Program Management • Module 1 • Why use statistics? • Why use the Environmental Results Program (ERP) as a case study? • Introduction to ERP and its measurement approach • Module 2 • Key concepts for planning and analyzing a single random sample • Orientation to ERP statistical spreadsheet tools • Group exercises and discussion • Module 3 • Key concepts for planning and analyzing comparisons of two random samples • Orientation to ERP statistical spreadsheet tools • Group exercises and discussion

3. Why Use Statistics? • Statistics are valuable whenever it’s too costly or inefficient to look at everything of interest– whether widgets or dry cleaners • Random samples of facilities provide a picture of everyone’s performance, with measurable uncertainty • Sample must be “representative” • E.g., a sample of volunteer facilities cannot be extrapolated to the population as a whole

4. Uncomfortable with Statistics? • That’s normal, but consider the alternatives • Census of all facilities • Accurate, but expensive • Visiting samples of facilities non-randomly • Cheaper, but little sense of how representative the data are

5. How to Use Statistics As Measurement Tool • Primary uses for statistics: • Estimate how well a group of facilities is performing at a particular point in time • Assess whether performance for a group of facilities has changed over time • Assess whether there are differences in performance between two different groups of facilities

6. What Gets Measured? • You can apply statistical analysis to just about any measurable characteristic of a group of facilities • Are they in compliance with an important requirement? With all requirements? • Are they implementing a key pollution prevention practice? • How much hazardous waste are they generating?

7. How Might This Help You? • Statistics can help you assess the need for (and efficacy of)... • traditional compliance approaches, • innovative environmental performance initiatives, • pollution prevention efforts, and • voluntary programs. • Why are you here? Do you think you might be able to use statistics in your work?

8. Why Use ERP As a Case Study? • ERP’s measurement approach can be used to evaluate group performance... • at a single point in time, • over time, and • across states • ERP itself involves elements of... • traditional compliance approaches, • innovative initiatives, • voluntary programs, and • pollution prevention

9. What is ERP? • Pioneered by MA DEP in 1990s • Printers, dry cleaners, photo processors • Integrates proven tools to cost-effectively improve performance in sectors characterized by large numbers of small sources of pollution • To date, average initial sector improvements from 5% to 30% • Improvement continues in later years • Stabilizes at high levels

10. ERP: Interlocking Tools, Integrated System

11. ERP: For Large Numbers of Small Pollution Sources • Developed by MA in 1990s for printers, dry cleaners and photo processors • Hundreds of facilities per sector • Significant aggregate environmental footprint • Traditional approaches untenable • DEP resources shrinking • ERP sectors have large numbers of facilities nationally • e.g., over 30,000 auto body/paint shops in U. S. • So far, six states with auto body ERPs

12. Current State of ERP • Endorsed for "scale up" by EPA Innovation Action Council (IAC) in 2000 • Based on MA results, NAPA evaluation and potential to address significant problems • Tangible EPA support since then • Grants, technical support, tools and resource flexibility • States ERP Consortium formed in 2006 • Organized as "forum" under ECOS • Promotes and supports use of ERP approaches • EPA NCEI is a steering committee member

13. Who's Doing ERP? (18 States, 8 EPA Regions) May 2007

14. ERP Now Covers 11 Sectors/Groups *FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

15. ERP Now Covers 11 Sectors/Groups (Cont.) *FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

16. How ERP Works: A Typical ERP Cycle Step 1:Inventory Step 2:Statistical BaselineInspections Step 3:Compliance Assistance Step 4:Self-Certification Step 7:InformedDecision-Making Step 6:Statistical Post-CertificationInspections Step 5:TargetedFollow-Up Renew Assistance and Certification (As Deemed Necessary)

17. How ERP Works: A Typical ERP Cycle Step 1:Inventory Identify the myriad small facilities that are sources of pollution, many of which are often unknown to regulators.

18. How ERP Works: A Typical ERP Cycle Step 2:Statistical BaselineInspections Conduct random inspections to accurately measure existing environmental performance and focus outreach on the biggest problems.

19. How ERP Works: A Typical ERP Cycle Step 3:Compliance Assistance Work with trade associations to create and provide plain-language, user-friendly assistance that improves compliance and promotes pollution prevention.

20. How ERP Works: A Typical ERP Cycle Facilities conduct self-assessments using a detailed checklist closely linked to assistance materials. Responsible officials certify to their facilities’ environmental performance on each item. If necessary, they submit plans to return to compliance. In some cases, certification has been mandatory; in others, voluntary. In all cases, compliance is required. Step 4:Self-Certification

21. How ERP Works: A Typical ERP Cycle Identify potential problem facilities via certification analysis, and target them for inspections, correspondence or phone calls. Provide assistance and/or initiate enforcement, as needed. Step 5:TargetedFollow-Up

22. How ERP Works: A Typical ERP Cycle Conduct random inspections to accurately estimate performance changes and verify facility certifications. Step 6:Statistical Post-CertificationInspections

23. How ERP Works: A Typical ERP Cycle Assess performance data and consider whether to adjust compliance assistance or other strategies directed at the sector or, if sufficient progress has been made over time, target resources elsewhere. Step 7:InformedDecision-Making Renew Assistance and Certification (As Deemed Necessary)

24. How ERP Measurement Works • Performance assessment is largely based upon random, inspector-collected data before and after self-certification • Data are representative and objective • Recognize that baseline inspections may have an effect on performance • It’s part of what you are measuring • Critical to follow principles of good data quality, such as: • Identify as complete a list of facilities as possible • Design a sound inspection checklist (garbage in, garbage out) • Train inspectors to follow sampling protocol & collect good and complete data • Quality Assurance Project Plans (QAPPs) help many states

25. Who Gets Inspected? • Random samples of all facilities • Typically about 35-50 per round of inspections, sometimes higher • Even in voluntary certification programs, you don’t just inspect the volunteers • Quality issues with just sampling volunteers: • missing the big picture, • self-selection bias, and • potential to miss spillover effects

26. What Gets Measured in ERP? • States have a lot of flexibility in choosing their measures, but there is convergence • Environmental Business Practice Indicators (EBPIs) are basic measurement building blocks • EBPIs represent what an agency considers to be the most important environmental aspects for the group • Typically multimedia, including both compliance and beyond-compliance issues • A multimedia inspection checklist may have well over 100 questions, but most states focus on 10-30 EBPIs • Statistical approaches can be used to infer how well an overall group is performing on EBPIs, based upon just the results from random inspections

27. EBPIs: Roll-up and Stand-Alone • Roll-up indicator • Summarizes the results of several questions • E.g., a Maryland EBPI tracked the implementation of several emission-reducing painting practices at auto body shops. Maryland observed an increase in use of such painting practices from 40% to 62% of inspected facilities. • Stand-alone indicator • Linked to one important question on the facility checklist • E.g., a Rhode Island auto body EBPI tracked the usage of methylene chloride, a dangerous paint-stripping chemical. Rhode Island observed a decrease in usage from 33% of inspected facilities to 5% of inspected facilities.

28. Other ERP Measures • For simplicity, statistical analysis of EBPIs is the focus of this training course • ERP states use statistics to calculate other important measures, too. E.g., • Group compliance score: Extent to which the average facility is achieving all relevant compliance-related EBPIs • Certification accuracy: Extent to which facility self-certification responses agree with observations of inspectors • Change in environmental condition: Extent to which the environment has changed, pollution has decreased, etc., since ERP implementation

29. Coming Up: A Closer Look at theTwo Main ERP Analyses • Module 2: Current state of performance • Looking at a single random sample • Module 3: Difference over time • Looking at 2 random samples, before and after certification • Difference between states is very similar, but not our focus (for simplicity)

30. For more information… Steve DeGabriele Director, Business Compliance Division Massachusetts DEP Steven.Degabriele@state.ma.us Work: 617-556-1120