Understanding Sampling Methods in Research

1. SOCY3700 Selected OverheadsProf. BackmanSpring 2008 Update history

2. Central Limit Theorem If repeated random samples of size n are drawn from any population with mean μ and standard deviation δ, the sampling distribution of sample means will be normal as n gets large, with mean μ and standard deviation δ/√n (also known as the standard error of the mean) . Hence, the standard deviation of the means drawn from many, many samples reflects 1) the standard deviation of the population, and 2) the sample size

3. Probability Sampling • Probability sampling is any method of drawing a sample of elements from a population such that the probability that any element or set of elements will be included in the sample is known and is not zero • The chief advantage of probability sampling is that the accuracy (or lack thereof) of estimates of population parameters from the sample can be estimated

4. Finite Populations and Sampling • Sampling error estimation depends on the Central Limit Theorem • The Central Limit Theorem applies to infinite populations • Infinite populations are easy to do in theory, but rare in practice • If you sample everyone in a finite population, the sampling error would be 0 • The closer you get to sampling everyone, the smaller your error should be • Central Limit Theorem says error is proportional to δ/√n

5. Finite Populations and Sampling, cont. • The finite population correction factor (fpc) takes into account the reduction in error you should get from sampling all or a large fraction of a finite population • The fraction of the population that is in the sample, n/N, is called the sampling ratio (f) • fpc = (N-n)/(N-1) ≈ (N-n)/N = (1 – f) • The standard error of the mean from a finite population (with simple random sampling) is√fpc * (δ/√n) • In practice, we ignore the fpc when the sampling ratio is less than 10%

6. Simple Random Sampling (SRS) • Frame – complete list of the survey population • Sample size – calculated based on desired precision of results • Selection rule – random selection without replacement • Estimate of population mean is the sample mean • Unbiased • s.e. = √fpc * (δ / √sample size)

7. Simple Random Sampling: Advantages and Disadvantages • SRS advantages • Samples are easy to draw • Samples are easy to use • Estimation of errors is “easy” • SRS disadvantages • Not always the lowest standard error method • Requires complete roster • Can be very expensive • Completing the frame may be expensive • Reaching geographically dispersed respondents may be expensive • May require large sample sizes to deal with rare population elements • Most elements in the sample will not be rare

8. Telephone Survey Sampling Frames There are four methods for sampling phone numbers for general population telephone surveys • Sample from the phone book • Problems with unlisteds • Random digit dialing • With known exchanges, use random number generator to come up with numbers • Many non-working numbers • Plus-one, minus-one • Sample from phone book, but add or subtract 1 from the number before dialing • Buy a list of known working numbers • Usually the best solution

9. Stratified Sampling • Frame • Usual SRS frame except broken into exhaustive, mutually exclusive groups • Requires knowledge ahead of time about how many elements in the population there are in each group • Each group is a stratum (plural strata) • Sample size - calculated based on desired precision of results • Calculations more complex than with SRS because there are more alternatives

10. Stratified Sampling (2) • Selection rules • Cases are drawn from each stratum • Cases within strata are drawn by SRS • Two alternatives for number drawn with each stratum • Proportionate to size – every element in the population has an equal chance of being drawn into the sample, regardless of stratum • Disproportionate – some strata will have a larger proportion of the sample than they will of the population

11. Stratified Sampling (3) • Proportionate sampling is technically known as probability proportionate to size selection, or PPS • Disproportionate sampling is non-PPS selection • Disproportionate sampling can be used to get enough “rare elements” into a sample to allow analysis of such elements with a reasonable level of confidence

12. Stratified Sampling (4) • Estimation of the mean • If proportionate to size selection is used, the sample mean is an unbiased estimate of the population mean • If disproportionate selection is used, weights must be used to obtain an unbiased estimate of the population mean • Standard error of the mean will ordinarily be lower than the standard error from a simple random sample of the same size • The more homogeneous the elements are within strata, the more efficient stratified sampling will be

13. Stratified Sampling: Advantages and Disadvantages (compared with Simple Random Sampling) • Advantages • Reduced standard errors of estimate over SRS • Can thus get the same precision as SRS with smaller sample size • If proportionate selection is used, unweighted sample statistics can be used to estimate population parameters • Disproportionate selection can be used to get sufficient numbers of members of rare populations • Disadvantages • Requires advanced knowledge about stratum sizes • Disproportionate selection requires use of weights in making estimates of parameters

14. Cluster Sampling • Most complex method. Often used in conjunction with stratification and SRS; this is called multi-stage sampling • Frame • Broken into groups called clusters • Complete frame is needed only for clusters that are selected • It is necessary to know the size of clusters that are not selected • Sample size – usually calculated based on explicit tradeoff between costs and precision of results • Calculations more complex than with SRS or stratification because there are more alternatives

15. Cluster Sampling (2) • Selection rules • A sample of the clusters is drawn by simple random sampling • Within each cluster either all the elements or a simple random sample of the elements are drawn • When possible, sample sizes within clusters are drawn proportionate to size • NOTE that in cluster sampling only some of the clusters are used, while in stratified sampling, all of the strata are

16. Cluster Sampling (3) • Estimation of the mean • If clusters and elements within clusters were drawn so that all elements in the population had equal probabilities of selection, the sample mean is an unbiased estimate of the population mean. This rarely is possible • In the likely case of unequal probabilities of selection, weights must be used to obtain an unbiased estimate of the population mean • Standard error of the mean will ordinarily be higher than the standard error from a simple random sample of the same size • The more heterogeneous the elements are within strata, the more efficient cluster sampling will be • To the extent possible, each cluster should be representative of the entire population

17. Cluster Sampling:Advantages and Disadvantages (compared with Simple Random Sampling) • Advantages • Cost control • In general, the only reason to use clustering is to reduce financial or time costs • Can be used with stratification of clusters to help control standard errors • If proportionate selection is used, unweighted sample statistics can be used to estimate population parameters • Disadvantages • Sampling consultant probably needed • Larger standard errors than with SRS • Parameter and error estimation usually requires use of weights

18. Sample Pathologies • Biggest, most common problem: non-response • Estimation of parameters and errors assumes that data were collected from every element in the sample • Limitations on generalizability due to mismatch between the population of interest (target population) and the frame (survey population) • Called coverage error

19. Surveys and the US Census The Census has numerous benefits for survey researchers • The decennial census is mostly a mail survey • Therefore, the Census Bureau sponsors a great deal of research on writing questions and other aspects of mail surveys • Census information is often used in developing stratified and cluster samples, where knowledge of population counts is necessary • Census information is often used to evaluate how well a sample covers a population • Especially when there is nonresponse, comparing demographic data from the sample with demographic data from the census can give some idea of who the nonrespondents were

20. Sampling Review • Rule of thumb sampling error of a proportion at the 95 percent confidence level = 1 / square root (sample size) • If size = 400, error = 1/20 = 5% • The Central Limit Theorem is important for social science research because it provides the mathematical basis for using probability samples 1) to make estimates of parameters from large populations using small samples and 2) to estimate the precision of those estimates

21. Sampling Review (2) • In both stratified and cluster sampling the survey population is divided into exhaustive, mutually exclusive groups. Each group could be either a stratum or a cluster • If we use all the groups in our final sample, we call each group a stratum • If we use only some of the groups in our final sample, we call each group a cluster

22. Dillman on the Survey Process • Dillman analyzes the survey process from an exchange theory perspective • There is an exchange between the researcher and the respondent • Compliance with researcher’s request for information is a function of the social rewards the researcher can offer the respondent • Rewards such as gratitude, opportunity to have a say on something important

23. Surveys á la Dillman:Eight Steps • Decide what information you need • Choose a survey method • Draw a sample • Write questions • Design the questionnaire • Field the survey • Turn answers into usable data • Report results Source: Patricia Salant and Don A. Dillman. 1994. How to Conduct Your Own Survey. NY: Wiley

24. Writing Survey Questions • Question topics • There is little you can’t ask about • Useful distinction: • Questions about subjective states like attitudes, beliefs, and knowledge • Questions about objective phenomena like behavior or demographic attributes • Always remembering that in a questionnaire even objective phenomena are filtered through the respondent’s mind

25. Writing Survey Questions (2):Question Form • Two basic question forms: open-ended and closed-ended • Open-ended questions are questions to which respondents can give any answer • Closed-ended questions both ask a question and provide the respondent with preset answers to the question to choose among Pp. 177ff in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

26. Writing Survey Questions (3):Closed-ended Questions • Questions with ordered categories • E.g., Likert scale items • When there is an order, be sure to use it • Questions with unordered categories • Partially closed-ended • One option is something like“Other (please specify) ____”

27. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts • Avoid jargon, slang, and abbreviations • Avoid ambiguity, confusion, and vagueness • Whatever • Avoid emotional language • Can evoke frames that effectively hijack the intent of the question • Avoid prestige bias Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

28. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts (2) • Avoid double-barreled questions • Do not confuse beliefs with reality • Avoid leading questions • Avoid asking questions that are beyond respondents’ capabilities Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

29. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts (3) • Avoid false premises • Avoid asking about intentions in the distant future • Avoid double negatives • Avoid overlapping or unbalanced response categories Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

30. Questionnaire Layout (1) • Very important • Reflects your professionalism in the eyes or ears of your respondents and the eyes of your interviewers • Affects the likelihood of measurement error through respondent or interviewer error • Affects response rate • In mail surveys designed primarily with respondent in mind • In telephone and face-to-face surveys, designed with both interviewer and respondent in mind

31. Questionnaire Layout (2):Mail Surveys • Overall objectives • Minimize perceived (and real) respondent burden • Don’t confuse respondent • Simplify later data entry • Make a booklet • Questions are enclosed inside a booklet made of folded legal sized (8.5 x 14 inch) paper • No questions on the front or back of the booklet

32. Questionnaire Layout (3):Mail Surveys • Front page of booklet: • Title of study • Some graphic stuff • Sponsor • Return address • Back page • Request for comments • Thank you • Return address and telephone contact information

33. Questionnaire Layout (4):Mail Surveys • Overall question sequence • Start easy • First question must grab attention, reflect the issues in the cover letter, and not be too difficult or threatening • Start on topic • Group like questions together • Makes writing transitions easier • Keep threatening questions until later in the questionnaire • Get your demographics last • That’s probably least important to you and apparently least relevant to respondent

34. Questionnaire Layout (5):Mail Surveys • Layout of individual pages • Use white space • What counts is not how many pages the survey is, but rather how long it seems to be to respondents • Use fonts consistently to distinguish questions, answers, and instructions • Dillman likes to use bold for questions, all caps for answers, unbolded for transitions, and unbolded in parentheses for instructions • Establish a vertical flow • Precode the answers, usually on the left margin

35. Fielding Mail Surveys (1) Overview 1. We’re always trying to increase response rates 2. Respondents are most likely to respond if they think benefits outweigh their costs 3. We need to keep respondents engaged from the opening of the mail through the returning of the completed questionnaire Source: Salant and Dillman

36. Fielding Mail Surveys (2) Bottom lines 1. Mail survey response rates depend very much on the number of contacts 2. Mail surveys require advanced planning - Be sure you have the resources to meet the schedule 3. What really matters is the overall look and feel of the questionnaire - It’s a lot like buying (or selling!) a car Source: Salant and Dillman

37. Fielding Mail Surveys (3) • First mailout – advancednoticeletter • Sent to the entire sample • Mailed first class • Handwritten signature • Explains why there will be a survey • Explains why participation will be appreciated • Put yourself on the mailing list for this and all other mailings

38. Fielding Mail Surveys (4) • Second mailout – cover letter, questionnaire, and return envelope • Sent one week after advanced notice • Cover letter • Personalized • Explains survey purpose • Explains ID# on the questionnaire and promises confidentiality • Reinforces importance of everyone’s participation • Specifies who should complete the questionnaire • Thanks respondent for participation • Hand signed Source: Salant and Dillman

39. Fielding Mail Surveys (5)[Second mailout, cont] • Questionnaire – with ID number • Return envelope is stamped, addressed, and ready for use

40. Fielding Mail Surveys (6) • Third mailout – postcardfollowup • 4 to 8 days later • Personalized • Reminding and thanking • Fourth mailout – new cover letter, questionnaire, and return envelope • Three weeks after the second mailout (the first one with a copy of the questionnaire) • Sent only to addresses that have not yet returned the survey

41. Fielding Mail Surveys (7) • The four mailings should yield a final response rate of 50 – 60 percent • To further increase response rate, one can: • Send another follow up like the fourth mailing • Send the follow up as certified or express mail • Telephone • Often you will discover that people shouldn’t have been in the sample in the first place

42. Experiments: Overview • Experiments are particularly important in microsociological research, i.e., social psychology • It is difficult to have enough control over the setting to do macrosociological experiments • We can do quasi-experiments • We can observe natural experiments • In an experiment we create a believable environment • In experiments we try to control “everything” • Ceteris paribus – all other things being equal

43. Levels of Involvement in Observational Research • There are several types of involvement of the researcher and the people he is trying to study • Outside observation – studying a group entirely from the outside, perhaps through intensive interviews • Overt complete observation – observing from within the group and known by the group to be there as an observer

44. Levels of Involvement in Observational Research (2) • Participating observer -- mostly an observer but also participating in the activities of the group • Observing participant -- mostly a group member but also observing the activities of the group • Participating observer and observing participant are types of participant observation

45. Street Corner Society: The Social Structure of an Italian Slum William Foote Whyte, 1943 (third edition, 1981)

46. Whyte Bio • Educated middle class upbringing • Loved to write • Attended Swarthmore in suburban Philadelphia • Engaged in some reform activities in college, but engaged even more in writing • Wrote a novel, decided it was lousy because he didn’t have enough to say • Got a Junior Fellowship at Harvard – three years just to hang around and do whatever research took his fancy (sort of)

47. The Research Problem • Whyte came to Harvard knowing mainly that he wanted to study slums and somehow improve the world • Social scientific literature was just beginning to appear. He read lots of it • Other folks at Harvard had done similar work and were developing some theoretical ideas about group process • One would not think one would go to a slum to study group process, but in the end that was a big part of what Whyte did • Many of the ideas Whyte when he started his work came to naught • “We set out on the frontiers of our personal knowledge and began exploring beyond those frontiers” (Whyte 1984:63)

48. “Cornerville” • In the usual fashion, Whyte gave his city and neighborhood a psuedonym. Cornerville refers to the slum, now known to be Boston’s North End. He called Boston “Eastern City.” • At the time (around 1937) Cornerville was suffering the effects of The Great Depression • Predominately Italian in a city whose big politicians were mostly Irish • Many residents spoke only Italian

49. Getting In • Wandered around Boston, settled on Cornerville because it “looked like” his vision of a slum • Could observe from the outside, but wanted to observe from the inside • After various failed schemes, introduced to Doc by the social worker in charge of girls’ programs at the local settlement house • Moved into the neighborhood

50. Doc • Doc (a psuedonym for Ernest Pecci) is probably the most famous informant in sociology • A pretty good sociologist himself for someone who never had a sociology course • Late 20s, mostly unemployed guy from the neighborhood • Informal leader of a group of similarly underemployed age mates • Interested in making things better