Principles of Good Data Collection

1. Principles of Good Data Collection Region V ERP Auto Body Training Chicago, IL November 18, 2009

2. Goals of Presentation • Develop understanding of quality issues connected with • Formulating measures • Designing questionnaire • Collecting data • Processing data

3. Step 1: Selection of Indicators • Purpose of Indicators • Types of Indicators • How to Formulate

4. Purpose of Indicators • Provide shorthand understanding of the performance of a facility (or group) at a point in time.

5. Different Types of Indicators • Activity-based: measures facility performance, (e.g., minimization plan written and posted where paint stripping is conducted?) • Outcome-based: measures quantitative environmental outcomes by estimating reductions in air emissions, hazardous waste, and materials use, (e.g., measure reduction in VOC emissions associated with use of low-VOC/water based solvents for all auto body shops.) • Regulatory: required practices, (e.g., filter capture efficiency rating of booth exhaust/filter system at least 98 percent) • Beyond-compliance: best management practices, (e.g., installed specialized controls (timers, motion sensors) that turn off or throttle back lights, heat, or equipment when areas are not occupied and/or in use)

6. Different Ways of Formulating Indicators • Quantitative v. Y/N questions • Single-item questions v. rolled-up questions

7. Example of Rolled-Up Indicators • Has your shop implemented all the required management practices that minimize emissions of MeCl use, (evaluating each application for alternative to MeCl; reducing exposure of MeCl strippers to air; optimizing conditions when using MeCl to reduce evaporation; and using proper storage and disposal techniques)?

8. Step 2: Questionnaire Design • Questionnaire Design Tips • Questionnaire Testing

9. Design Tip: Avoid Double Negatives • Unclear:Does the facility avoid using unapproved spray booths/stations to coat miscellaneous parts or products or vehicle subassemblies? • Explicit:Does the facility only use approved spray booths/stations that have a full roof, at least three complete walls or side curtains, and is ventilated so that air is drawn into the booth, to coat miscellaneous parts or products or vehicle subassemblies?

10. Design Tip: Definite Time Frames • Unclear:Does the facility maintain records of MeCl use for paint stripping? • Explicit: Does the facility have records of annual usage of MeCl for paint stripping?

11. Design Tip: Consider Measurability • “Are employees aware of P2 practices?” – difficult to observe • “Do employees receive documented P2 training at least once a year?” – better • "Are filters changed regularly to ensure good airflow?” -- even better

12. Before Using Questionnaires in the Field… Test them! • This training is the last “test” before we go live • Testing helps to ensure that questions are interpreted the same way (precision and comparability) • Testing helps identify any errors or ambiguities (precision and completeness)

13. Step 3: Data Collection • Who Collects the Data? • Tips for Inspector Training • Timing Considerations • Electronic v. Paper

14. Who Is to Answer Questionnaire? • SBEAP Field Observers • EPA Inspectors • Facilities (self-reporting) Each option can be evaluated in terms of quality

15. Who Is to Answer Questionnaire? • SBEAP Field Observers • May have less experience conducting inspections • Has no authority to enter facilities (will have to request entry) • Follow-up action in the form of assistance

16. Who Is to Answer Questionnaire? • EPA Inspectors • May have more experience conducting inspections • Has authority to enter facilities • Can take follow-up action in the form of informal or formal enforcement

17. Who Is to Answer Questionnaire? • Facility self-reporting • Raises bias concerns • Training less reliable Note: In ERPs in various sectors (dry cleaning, printing, farming, etc) around the country, verification inspections by regulators show that self-reporting starts at reasonable accuracy rates (~70%) and improves in successive self-reporting cycles.

18. Musts for Data Collection Train field observers so that they... • Interpret questions consistently • Understand and follow sampling protocol • Answer all questions (no blanks) • Always use ink • Use consistent protocol for corrections • e.g., double-strikeout plus initials (will discuss protocol in Planning for Site Visits presentation)

19. More on Consistent Interpretation • Interpretation by field observer • e.g., are observers looking for faults and violations, or trying to gather a holistic impression of facility performance, or both? • Training observers together can help align approach • But there will be differences and potential biases • e.g., more violations in one state might mean simply that those observers had a more strict interpretation of compliance. • Best strategy is to make indicators as specific and explicit as possible.

20. Timeframe of Data Collection • Short timeframe desirable, E.g., 2-3 months • Consider seasonality issues • VOC usage – may not be same peak for each shop • Not all states, or regions of all states, are in same snow belt. May impact prime season for collisions Reminder: Project goal to complete baseline inspections by first quarter 2010 (Jan 1st –March 31st).

21. Step 4: Data Processing • Data Entry Options • Certification of Data Quality

22. Data Entry Options for Checklists • Transfer data from checklist into electronic format, i.e. using on-line survey developed by WI SBEAP Project Lead. • Scan checklists and send electronic copy (PDF) to WI SBEAP Project Lead. • Mail hard “copies” of completed checklists to WI SBEAP Project Lead for data entry. • Some SBEAPs may not want to keep records of shop-specific data in order to preserve their confidentiality and assistance stance. However, to protect against lost mail, make copies of checklists and destroy after they are electronically filed by WI SBEAP Project Lead.

23. Data Entry Tip If field observers transfer checklists into electronic survey, do so as soon as possible after inspection

24. Certification of Data Quality • Each state lead will be responsible for the completeness of their state’s data. • Copies of completed checklists will be sent to Project Lead/Quality Assurance Officer (WI SBEAP) along with a “Certification of Data Quality” signed by each state project. • Certification of Data Quality Statement will be provided to each state lead prior to data collection. • Please send completed checklists and certification statement all at once to minimize error. • Statement will look something like this: I, …………..(project lead), from the state of ……….., certify that the enclosed field observation checklists meet the data quality standards described in the Region V ERP Auto Body Training Workshop on November 18-19 2009.

25. For more information • Tara Acker, NEWMOA • taraacker@gmail.com • (413) 549-5309 • or • Renee Bashel, WI Department of Commerce • renee.bashel@wisconsin.gov • (608) 264-6153