Design Method of Data Collection

Surveys and Polls Observational Studies Experimentation Design Method of Data Collection

Sampling/Chance/ Random Errors Nonsampling Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects Sampling

sample population Sources of Nonsampling Errors Selection bias Population sampled is not exactly the population of interest. e.g. KARE 11 poll, telephone interviews

Sources of Nonsampling Errors Non-response bias People who have been targeted to be surveyed do not respond. Non-respondents tend to behave differently to respondents with respect to the question being asked.

1936 U.S. Election • Country struggling to recover from the Great Depression • 9 million unemployed • 1929-1933 real income dropped by 1/3

This man has never been President. 1936 U.S. Election • Candidates: • Franklin D Roosevelt (Democrat) • Deficit financing - “Balance the budget of the people before balancing the budget of the Nation” • Albert Landon (Republican) • “The spenders must go!”

1936 U.S. Election • Roosevelt’s percentage • Digest prediction of the election result • Gallup’s prediction of the Digest prediction • Gallup’s prediction of the election result • Actual election result 43% 44% 56% 62% • Digest sent out 10 million questionnaires to people on club membership lists, telephone directories etc. • received 2.4 million responses • Gallup Poll used another sample of 50,000 • Gallup used a random sample of 3,000 from the Digest lists to predict Digest outcome

Sources of Nonsampling Errors Self-selection bias People decide themselves whether to be surveyed or not. Much behavioural research can only use volunteers.

Sources of Nonsampling Errors

This poll is not scientific and reflects the opinions of only those Internet users who have chosen to participate. The results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole. The QuickVote sponsor is not responsible for poll content, functionality or the opinions expressed therein. Sources of Nonsampling Errors

Sources of Nonsampling Errors Question effects Subtle variations in wording can have an effect on responses. e.g. “Should euthanasia be legal?” vs. “Should voluntary euthanasia be legal?”

18 August 1980 New York Times/CBS News Poll “Do you think there should be an amendment to the constitution prohibiting abortions?” Yes 29% No 62% Later the same people were asked: “Do you think there should be an amendment to the constitution protecting the life of the unborn child?” Yes 50% No 39%

Ethnicity 1986 1991 1996 94.6 94.3 Single 81.0 Ethnicity 81.2 78.1 European 65.8 9.1 9.6 Maori 7.6 Two Ethnicities 4.0 4.5 11.2 2.9 2.7 European & Maori 4.7 0.0 0.6 Two European gps 4.5 Question Effects in the NZ Census 1986: “What is your ethnic origin? (Tick the box or boxes which apply to you.) 1991: “Which ethnic group do you belong to?” (Tick the box or boxes which apply to you.) 1996: “Tick as many circles as you need to show which ethnic group(s) you belong to.”

Sources of Nonsampling Errors Interviewer effects Different interviewers asking the same question can obtain different results. e.g. the sex, race, religion of the interviewer

Interviewer Effects in Racial Questions In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked: “Do you personally feel that you trust most white people, some white people or none at all?” • White interviewer: 35% answered “most” • Black interviewer: 7% answered “most”

Sources of Nonsampling Errors Behavioural considerations People tend to answer questions in a way they consider to be socially desirable. e.g. pregnant women being asked about their drinking habits

Behavioural Considerations in Election • Official vote counts show that 86.5 million people voted in the 1980 U.S. presidential elections. • A census bureau survey of 64,000 households some weeks later estimated 93.1 million people voted.

sample Minneapolis Winona Sources of Nonsampling Errors Transferring findings Taking the data from one population and transferring the results to another. e.g. Minneapolis opinions may not be a good indication of Winonan opinions.

Sources of Nonsampling Errors Survey-format effects e.g. question order, survey layout, interviewed by phone or in- person or mail.

Sampling/Chance/ Random Errors Nonsampling Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects Sampling

1999 New Zealand Newspaper Graphic

Nonsampling Errors What nonsampling errors could be present if the results of the survey were used as representative of the views held by New Zealanders in general? Discuss in groups for 1 minute.

Nonsampling Errors • Selection bias • only those who read this paper • only those that can afford to pay for the call • Question effects • how do you answer the question?? • Self-selection bias

Survey Errors Nonsampling Errors Sampling/Chance/ Random Errors

Sampling/Chance/Random Errors • errors caused by the act of taking a sample • have the potential to be bigger in smaller samples than in larger ones • possible to determine how large they can be • unavoidable (price of sampling)

Nonsampling Errors • can be much larger than sampling errors • are always present • can be virtually impossible to correct for after the completion of survey • virtually impossible to determine how badly they will affect the result • must try to minimize in design of survey (use a pilot survey etc.)

Surveys/Polls A pilot survey is a small survey that is carried out before the main survey and is often used to identify any problems with the survey design (such as potential sources of non-sampling errors).

Surveys/Polls A report on a sample survey/poll should include: • target population (population of interest) • sample selection method • the sample size and the margin of error • the date of the survey • the exact question(s) being asked

Design Method of Data Collection