Quality Control for Field Operations

1. Quality Control for Field Operations

2. Overview • Goal • To ensure the quality of survey field work • Purpose • To detect and deter interviewer errors and falsification

3. Overview(cont.) What do we check in QC for field work? • Interviewer procedures • Forms • Interviewer productivity • Interviews • Address listing

4. Overview(cont.) What are the tools for performing QC on field work? • Interviewer observations • Formal / Informal reviews • Monitoring reports • Reinterview • Address listing QC

5. Interviewer Observations What are the goals of interviewer observations? • To assess effectiveness of interviewer training • To ensure interviewers understand and follow prescribed interviewing procedures • To provide interviewers with feedback to encourage continuous improvement

6. InterviewerObservations(cont.) What are interviewer observations? • Supervisor “shadows” interviewer during initial assignments • Observe interviewer’s work to ensure appropriate procedures are followed • Evaluate interviewer’s work and provide positive and negative feedback to interviewer • See Handout on Field Control Form Information

7. Interviewer Observations(cont.) What are the results of observations? • Supervisor decides if interviewer’s performance is acceptable • If so, allow interviewer to continue work independently • If not, re-train interviewer on procedures or release interviewer from the operation

8. Formal / Informal Reviews What are the goals of reviews? • To ensure the quality of the data collected on forms and questionnaires • To ensure interviewers and clerks understand and follow procedures for completing forms and questionnaires

9. Formal / Informal Reviews(cont.) How are informal reviews performed? • Supervisor meets with employees to review forms and/or questionnaires • Review for legibility, completeness, and accuracy • Provide positive and negative feedback to employee for continuous improvement • Repair errors where necessary

10. Formal / Informal Reviews(cont.) How are formal reviews performed? • Usually performed in a field office by office staff • Review questionnaires or listings for legibility, completeness, and accuracy • Repair errors or return materials to the field when necessary

11. Formal / Informal Reviews(cont.) What work products are subject to reviews? • Any form or questionnaire that requires a quality check (e.g., payroll forms, questionnaires, address listing forms, etc.)

12. Formal / Informal Reviews(cont.) What are the results of informal reviews? • Supervisor determines if the form or questionnaire meets quality standards • If so, the form/questionnaire is accepted and submitted • If not, repair the form/questionnaire prior to submission

13. Monitoring Reports What are the goals of monitoring reports? • To track status of operations • To inform management of progress • To identify problems early for timely rectification

14. Monitoring Reports(cont.) How do we know what reports to monitor for QC purposes? • Define quality goals for the operation (e.g., response rates, productivity rates, cost, etc.) • Identify variables that provide information on quality goals • Design report(s) that present the important variables for quick review and assessment

15. Monitoring Reports(cont.) How do we monitor the reports for QC purposes? • Determine necessary frequency for monitoring (usually daily) • Look for outlier values (e.g., low response rates, high costs, etc.) • Determine reason(s) for outlier values • Take appropriate action

16. Reinterview What is reinterview? • Quality check on interviewing activities • Primary tool for ensuring quality of interviewers’ field work • Independent, objective review of interviewing work

17. Reinterview(cont.) What types of reinterview can be used? • Random reinterview • Administrative reinterview • Supplemental reinterview

18. Reinterview(cont.) What decisions are made in reinterview? • Falsification (release interviewer) • Interviewer error (retraining/feedback) • Respondent error (no action) • Rework may be necessary for falsification or other serious errors

19. Random Reinterview What is the purpose of the random reinterview component? • Provides broad protection over entire workload and all interviewers • Serves as an effective deterrent for interviewer falsification

20. Random Reinterview(cont.) What is the sampling design for the random reinterview component? • Select random sample of cases within every enumerator’s workload, or • Two-stage design: take a sample of interviewers, then a sample of cases within those interviewers’ workloads

21. Administrative Reinterview What is the purpose of the administrative reinterview component? • Provides “targeted” approach to identifying suspect work • Provides valuable performance data to field supervisors

22. Administrative Reinterview(cont.) How does the administrative model work? • Accumulates data on work characteristics for all interviewers • Vacant housing units • Partial interviews • Single-person households • Deleted housing units • Average population per housing unit

23. Administrative Reinterview(cont.) How does the administrative model work? (cont.) • Compare accumulated data for each interviewer to the data for all other interviewers in their assignment area • Identify and flag interviewers who are significantly different than their area average

24. Administrative Reinterview(cont.) How does the administrative model work? (cont.) • Provide field supervisors with a list of “outlier” interviewers and what variables caused the flag • Field supervisors determine if the interviewers’ “outlier” status is explicable • More of suspect interviewer’s cases put into reinterview

25. Supplemental Reinterview What is the purpose of the supplemental reinterview? • Provides field supervisors with the capability of checking work for selected interviewers • Tool for use during investigation of suspected falsification

26. Supplemental Reinterview(cont.) What is the sampling design for the supplemental reinterview? • Select a random sample of cases from the interviewer’s workload • Field supervisor can specify how many cases to check

27. Address Listing QC What is the goal of address listing QC? • To ensure the quality of address listing work • To protect against undercoverage and overcoverage

28. Address Listing QC(cont.) What are the tools for performing address listing QC? • Dependent QC • Advance listing • Suppression

29. Address Listing QC(cont.) What decisions are made in address listing QC? • If errors exceed acceptable limit, recanvass assignment area • Repair minor mistakes • Listers may be released or retrained depending on severity of errors

30. Dependent QC What is dependent QC? • Random, independent check of addresses listed by survey listers • Field supervisors verify existence of housing units or validity of housing unit deletes

31. Dependent QC(cont.) What is the sampling design for dependent QC? • Select a random sample of housing units to verify from the listers’ workload • Sample should include all types of listings (e.g., verifications, adds, deletes)

32. Advance Listing What is advance listing? • Field supervisor does initial listing of assignment areas prior to making assignments to listers • Gives supervisor knowledge of the area so they are able to detect faulty or inadequate work

33. Suppression What is suppression? • Selected addresses are purposely left off listings • Listers’ ability to find and add the suppressed addresses is assessed • Provides confidence in the listers’ ability to perform adequate canvassing