Non-sampling errors

1. Non-sampling errors (Session 20)

2. Learning Objectives By the end of this session, you will be able to • describe the types of non-sampling errors that arise in survey work • explain actions that may be taken to minimise commonly occurring non-sampling errors • have a greater appreciation that sampling errors is only a small component of all errors that may arise and that close attention to reducing non-sampling errors is equally or more important in survey work.

3. Non-sampling errors: 1 Non-sampling errors cover all errors other than those due to sampling a subset of the population. In both surveys and censuses, it is quite usual to find non-sampling errors because their absence implies that the data collection process has been: • Implemented and enumerated perfectly, & • Completely free of measurement errors, i.e. inaccuracies in the recording of information from selected units.

4. Non-sampling errors: 2 Non-sampling errors are not all due to avoidable mistakes and/or deficiencies. They can often occur because of decisions by the researchers to balance the need for good quality data with the need to obtain timely data at acceptable cost. The problem then reduces to one of defining and minimising errors associated with the data collection and data processing procedures.

5. Types of non-sampling errors Non-sampling errors can be of various types • Coverage (or Frame) errors • Non-response errors • Measurement errors • Data handling errors Note that the first more often applies to sample surveys, while the last three apply to both surveys and censuses.

6. Coverage (frame) errors In surveys, the sample is selected from a list, i.e. a sampling frame, of all population members. An inadequate frame leads to coverage errors. Often can have either • under-coverage(missing elements), or • over-coverage (duplicates) Both lead to biased results. See below for an example.

7. Minimising frame errors For under-coverage, consider re-defining the population, i.e. the target population is simply considered as the population which can be accessed by the frame. For duplicates, develop a system to identify the duplicates, e.g. by using additional information on the recording unit. Both under & over-coverage are minimised by using up-to-date frames, e.g. in UK the Postcode Address File is updated every 3 months, and is hence often used by the Office of National Statistics (ONS).

8. Non-response errors Non-response errors are all errors arising from: • Unit non-response, i.e. failure to obtain information from a pre-chosen sampling unit or population unit • Item non-response, i.e. failure to get a response to a specific question or item in the data recording form.

9. Types of non-response errors Discussion: What are the typical forms of non-response (both unit and item non-response) you encounter in your work? What are the reasons for non-response? How can such non-response errors be minimised?

10. Measurement Errors Measurement errors arise when the recorded response differs from the true value. They can occur for a variety of reasons, e.g. • by respondent (e.g. heads of households) giving an incorrect answer • because of instrument or question error • by interviewer error. Further, errors may be greater for some sub-groups of the population, e.g. those less literate, or those unwilling to co-operate.

11. Reasons for respondent errors Respondent errors arise for many reasons e.g. • respondent gives an incorrect answer, e.g. due to prestige or competence implications, or due to sensitivity or social undesirability of question • respondent misunderstands the requirements • lack of motivation to give an accurate answer • “lazy” respondent gives an “average” answer • question requires memory/recall • proxy respondents are used, i.e. taking answers from someone other than the respondent. How can such errors be minimised?

12. Instrument Errors Instrument or question errors arise when • The question is unclear, ambiguous or difficult to answer • the list of possible answers suggested in the recording instrument is incomplete • requested information assumes a framework unfamiliar to the respondent • the definitions used by the survey are different from those used by the respondent (e.g. how many part-time employees do you have? See next slide for an example) How can such errors be minimised?

13. An example of instrument error The following example is from Ruddock (1998) – see slide 18 In the Short Term Employment Survey (STES) conducted by Office of National Statistics in UK, data are collected on numbers of full-time and part-time employees on a given reference date. Some firms ignored the reference date and gave figures for employees paid at the end of the month, thus including those who joined and those who left in that month – leading to an over-estimate. Firms found it difficult to give details of part-time employees as their definition of “part-time” did not agree with that used by ONS.

14. Interviewer errors Interviewer errors arise when • different interviewers administer a survey in different ways • differences occur in reactions of respondents to different interviewers, e.g. to interviewers of their own sex or own ethnic group • inadequate training of interviewers • inadequate attention to the selection of interviewers • there is too high a workload for the interviewer How can such errors be minimised?

15. Data handling errors Data handling errors can occur from the stage of data collection up to the final stages of data analysis. Types of errors that can arise include:- • errors in transmission of data from the field to the office • errors in preparing the data in a suitable format for computerisation, e.g. during coding of qualitative answers • errors in computerisation of the data • errors during data analysis, e.g. imputation and weighting. Do any of these types of error occur in your work. If so, what can you do to minimise them?

16. Measuring non-sampling errors Measuring non-sampling errors is difficult and often impossible. Attempts have often been through specific additional studies, e.g. characteristics of non-respondents in the 1996 British Crime Survey were investigated by a mini-questionnaire to those living in 25% of non-responding addresses. Several studies to assess non-sampling errors can be found in Ruddock (1998) (see slide 18 for full ref.) & in Lessler, J.T. and Kalsbeek, W.D. (1992) Non-sampling error in surveys; Wiley.

17. Non-sampling errors: Key Points Non-sampling errors are inevitable in production of national statistics. Important that:- • At planning stage, all potential non-sampling errors are listed and steps taken to minimise them are considered. • If data are collected from other sources, question procedures adopted for data collection, and data verification at each step of the data chain. • Critically view the data collected and attempt to resolve queries immediately they arise. • Document sources of non-sampling errors so that results presented can be interpreted meaningfully.

18. References Ruddock, V. (1998) “Measuring and Improving Data Quality” UK Govt. Statistical Service Methodology Series No. 14, for a very comprehensive coverage of non-sampling errors. This document may be downloaded from http://www.statistics.gov.uk/methods_quality/publications.asp Lepkowski, J. (2004) Non-observation error in household surveys in developing countries. Chapter VIII, pp 149-169 of the UN Publication An Analysis of Operating Characteristics of Household Surveys in Developing and Transition Countries: Survey Costs, Design Effects and Non-Sampling Errors. Available athttp://unstats.un.org/unsd/hhsurveys/index.htm

19. Practical work follows…