Chapter 1 Section 4

1. Chapter 1Section 4 Sources of Error In Sampling

2. 1 Chapter 1 – Section 4 • Learning objectives • Understand how error can be introduced during sampling

3. Chapter 1 – Section 4 • We will have errors when we calculate from a sample • Some errors can be minimized • Poor design of the sampling frame • Poor design of the sample questions • Some errors are difficult to control in sampling • Incomplete information compared to the entire population

4. Chapter 1 – Section 4 • One type of error, samplingerrors, occur because we use only part of the population in our study • One type of error, samplingerrors, occur because we use only part of the population in our study • Samples consist of only part of the total data • One type of error, samplingerrors, occur because we use only part of the population in our study • Samples consist of only part of the total data • Samples are usually more realistic to analyze • One type of error, samplingerrors, occur because we use only part of the population in our study • Samples consist of only part of the total data • Samples are usually more realistic to analyze • Because there are individuals in the population that are not in our sample, sampling errors are difficult to control • One type of error, samplingerrors, occur because we use only part of the population in our study • Samples consist of only part of the total data • Samples are usually more realistic to analyze • Because there are individuals in the population that are not in our sample, sampling errors are difficult to control • We will study sampling errors in future chapters

5. Chapter 1 – Section 4 • Another type of error, nonsamplingerrors, occur from the actual survey process • Preference is given to selecting some individuals over others • Individual answers are not accurate (for various reasons) • Nonsampling errors can often be controlled or minimized with a well-designed survey and sampling technique

6. Chapter 1 – Section 4 • Types of nonsampling error • Types of nonsampling error • Using an incomplete frame • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Misrepresented answers • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Misrepresented answers • Data checks • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Misrepresented answers • Data checks • Questionnaire design • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Misrepresented answers • Data checks • Questionnaire design • Wording of questions • Types of nonsampling error • Using an incomplete frame • Individuals who respond have different characteristics than individuals who do not respond • Interviewer errors • Misrepresented answers • Data checks • Questionnaire design • Wording of questions • Order of questions, words, and responses

7. Chapter 1 – Section 4 • Using an incomplete frame • If the sample is not representative of the population, then the descriptive statistics of the sample may be very different from the true population parameters • This is similar to performing a stratified sample but with the incorrect strata percentages

8. Chapter 1 – Section 4 • The LiteraryDigest used their polls to predict the winner of presidential elections • The LiteraryDigest used their polls to predict the winner of presidential elections • Their previous polls were accurate • The LiteraryDigest used their polls to predict the winner of presidential elections • Their previous polls were accurate • In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide • The LiteraryDigest used their polls to predict the winner of presidential elections • Their previous polls were accurate • In 1936, the Literary Digest predicted that Alf Landon would defeat Franklin Roosevelt in a landslide • In the actual election, Roosevelt won in a landslide

9. Chapter 1 – Section 4 • Why was the LiteraryDigest so far off? • The 1936 frame was not representative of the total voting population • Why was the LiteraryDigest so far off? • The 1936 frame was not representative of the total voting population • The sampling process was not completely random • Why was the LiteraryDigest so far off? • The 1936 frame was not representative of the total voting population • The sampling process was not completely random • The frame had too large of a proportion of Republicans, who generally favored Landon • Why was the LiteraryDigest so far off? • The 1936 frame was not representative of the total voting population • The sampling process was not completely random • The frame had too large of a proportion of Republicans, who generally favored Landon • The frame had too small of a proportion of Democrats, who generally favored Roosevelt • Why was the LiteraryDigest so far off? • The 1936 frame was not representative of the total voting population • The sampling process was not completely random • The frame had too large of a proportion of Republicans, who generally favored Landon • The frame had too small of a proportion of Democrats, who generally favored Roosevelt • Republicans were overrepresented and Democrats were underrepresented

10. Chapter 1 – Section 4 • Some individuals selected in the sample may not respond • People who do not respond may feel differently than people who do respond • This can be controlled with callbacks and incentives • An extreme case of this error occurs in self-selected samples

11. Chapter 1 – Section 4 • Interviewer errors may occur when • The interviewer has a vested interest in the results • Interviewer errors may occur when • The interviewer has a vested interest in the results • The interviewer is not trained to obtain accurate information • Interviewer errors may occur when • The interviewer has a vested interest in the results • The interviewer is not trained to obtain accurate information • The individuals feel pressure or an obligation to provide an answer that the interviewer desires • Interviewer errors may occur when • The interviewer has a vested interest in the results • The interviewer is not trained to obtain accurate information • The individuals feel pressure or an obligation to provide an answer that the interviewer desires • For example, if your server watches you when you fill out the restaurant’s service satisfaction questionnaire …

12. Chapter 1 – Section 4 • Some individuals may deliberately lie in their responses • Even if the other aspects of the survey are designed well, the individual may want to “look good” or may be embarrassed about giving truthful answers

13. Chapter 1 – Section 4 • Data recorded by the individuals may be entered incorrectly onto the questionnaire • Data recorded by hand can often be entered into the system incorrectly (“typos”) • There are some techniques for detecting these errors, such as examining outliers

14. Chapter 1 – Section 4 • Sometimes the questionnaire design affects the response • Sometimes the questionnaire design affects the response • An openended question is one where the respondent is free to choose the response • Sometimes the questionnaire design affects the response • An openended question is one where the respondent is free to choose the response • A closedended question is one where the respondent must choose one in a predetermined list of responses • Sometimes the questionnaire design affects the response • An openended question is one where the respondent is free to choose the response • A closedended question is one where the respondent must choose one in a predetermined list of responses • Each has their advantages

15. Chapter 1 – Section 4 • Open ended questions • Allows the respondent to choose their own answer • Gives the flexibility to represent a variety of options • Open ended questions • Allows the respondent to choose their own answer • Gives the flexibility to represent a variety of options • Closed ended questions • Limits the number of possible responses, making the analysis easier • Gives the respondents a structure explaining more of the meaning and purpose of the question • Open ended questions • Allows the respondent to choose their own answer • Gives the flexibility to represent a variety of options • Closed ended questions • Limits the number of possible responses, making the analysis easier • Gives the respondents a structure explaining more of the meaning and purpose of the question • A combination of open ended and closed ended questions could be effective

16. Chapter 1 – Section 4 • Wording of questions could affect the accuracy of responses • Some questions are not asked in balanced form • Some questions are worded in confusing ways • Some questions are too vague • Wording of questions could affect the accuracy of responses • Some questions are not asked in balanced form • Some questions are worded in confusing ways • Some questions are too vague • The same question, worded in two different confusing ways, could result in two different results

17. Chapter 1 – Section 4 • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

18. Chapter 1 – Section 4 • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

19. Chapter 1 – Section 4 • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?”

20. Chapter 1 – Section 4 • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” • What will students say? • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” • What will students say? • May tend to say “yes” to both questions, even though they are complete opposites • Two questions for students • Question 1 – “Do you feel that no final exams should not be required to not last more than 2½ hours?” • Question 2 – “Do you feel that no final exams should not be required to not last less than 2½ hours?” • What will students say? • May tend to say “yes” to both questions, even though they are complete opposites • May tend to say “yes” because both have the words “no” and “required” and “2½ hours”

21. Chapter 1 – Section 4 • The order of questions, words, and responses can affect the responses • The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A • The order of questions, words, and responses can affect the responses • The responses to two questions may differ if they are ordered as Question A then Question B, as compared to if they are ordered as Question B then Question A • One example is when many unfamiliar candidates are running for a position … the order of listing their names affects the number of votes they get • The order of names is often drawn at random

22. Summary: Chapter 1 – Section 4 • There are reasons why samples have errors • Sampling errors are difficult to control because we use only part of the total population • Nonsampling errors are often largely avoidable with a good study design • Minimizing nonsampling errors is of high priority in designing a sample survey