International Standards for Survey Quality: A Total Survey Error Solution?

1. International Standards for Survey Quality:  A Total Survey Error Solution? Paul P. Biemer Lars E. Lyberg

2. Aside…Estimator of the MSE Using a Gold Standard is biased is unbiased (gold standard) See equation 2.20, Chapter 2,Overview of Design Issues: Total Survey Error in Marsden, P. and Wright, J. (2010). Handbook of Survey Research, Emerald Publishers

3. The Premise of this Paper • There are many standards for surveys; examples include • ISO 20252 for Market, Opinion and Social Research • OMB standards • NCES statistical standards • Quality guidelines developed by specific organizations (Stat Can; RTI; etc.) • ESS Code of Practice • Do these standards help in reducing total survey error for surveys? • The answer to this question is not clear.

4. Definitions: policies, standards, guidelines and recommended practices. • standards - A practice that is followed almost without exception. Deviations are not recommended and require approval of senior management. Corrective action should be taken when a standard is not being met.” • policies (no exceptions under any circumstances) • guidelines (can be skipped for “good” reasons), and • recommended practices (promoted but not obligatory) – Colledge and March (1997)

5. What are survey standards? • A document that • describes methods and procedures for the collecting, processing, storing, and presenting survey data.   • define the (minimal) level of quality and effort that is acceptable for all survey processes • May address only some survey processes; e.g., standards for writing survey reports • Should be auditable; vague standards have no effect on survey quality

6. What purposes do survey standards serve? • Communicates a level of quality that organizations establish as minimal for all surveys. • Provide consistency across surveys in different organizations • Facilitate communication of complex concepts, formulas, procedures and methodologies • Provide transparency of the methodologies used to produce a survey data set. • Transfers skills and knowledge of best survey practice

7. Some Problems with Survey Standards • Comprehensive standards can overwhelm survey practitioners • Unless mandated and audited, they are largely ignored. • Must be continually updated. • May conflict with client requirements. • Can be costly to develop, maintain and enforce. • No documented evidence exists that they improve data quality.

8. Idea advocated in this presentation • General survey standards have their strengths and weaknesses, but their effects on data quality have not be documented • Instead we advocate Standards for Survey Quality Investigations (SSQIs) • SSQI’s are like survey standards in that they describe methods and procedures. • SSQIs focus on methodological studies and survey quality investigations. • Example: the AAPOR standard for computing response rates.

9. What purposes would SSQIs serve? • To promote data quality investigations. • To encourage sharing of the results of survey methodological studies, including the data themselves. • To communicate the methods and the quality metrics used. • To facilitate methodological meta-analysis and comparative analysis • To increase transparency and consistency • To share skills and knowledge regarding survey quality investigations. • Decrease process variation

10. Some areas where SSQIs would be beneficial • Computation of frequently used quality indicators • weighted and unweighted response rates (e.g., AAPOR) • eligibility rates • tracing rates • frame coverage, duplication, and erroneous inclusion rates • design effects (unequal weighting and clustering) • keying, coding and editing error rates • Item nonresponse rates • reliability measures

11. Some areas where SSQIs would be beneficial (cont’d) • Reporting standards for these quality indicators • Best methods for collecting data to compute these quality indicators • Best practices for • Nonresponse followup studies, including two phase sampling methods • Investigations of interviewer variance • Evaluations of frame coverage bias • Reinterview studies • Experimentation with incentives

12. Resurrecting an old ITSEW idea: Survey Quality Wiki • “Wiki” – Hawaiian for “fast;” “backronymed” to stand for “what Iknow is” • Website that allows the easy creation and editing of any number of interlinked web pages via a web browser using a simplified text editor • Typically powered by “wiki software”to create collaborative websites for knowledge management. • Users use their editorial rights to remove material that is considered "off topic." Such is the case of the collaborative encyclopedia Wikipedia.

13. Questions for Discussion • How does the group feel about survey standards in general? • How well do they serve their stated purposes? • Are there good examples of cases where they have improved survey quality? • How does the group feel about SSQIs? Would such standards benefit survey quality research? • What types of SSQIs would be the most beneficial for survey research? • How should SSQIs be promulgated? Is a Survey Quality Wiki a good idea? Is there interest in developing it?