SOCY3700 Selected Overheads Prof. Backman Fall 2007

1. SOCY3700 Selected OverheadsProf. BackmanFall 2007 Update history 10/05/07 Add slides 1-25 10/10/07 Add slides 26-33 10/14/07 Add slides 34-45

2. Measurement Validity • Measurementvalidity is the extent to which a measure measures whatever it is intended to measure • Three types of measurement validity • Facevalidity – does the measure seem (“on its face”) like it measures what it’s supposed to (often tested by asking experts and others)

3. Measurement Validity, cont. • Contentvalidity - the extent to which the measure covers the full range of the concept • The richer the concept (say, religiosity or feminism), the more likely that multiple indicators will be needed • Criterionvalidity – the extent to which the measure is supported by other accepted measures • Concurrent validity – how well the measure correlates with other measures of the concept • Predictive validity – how well the measure correlates with other concepts its should be related to

4. Levels of Measurement • Nominal – values identify categories only • Do not have arithmetic meaning • Also called categorical variables • When there are only two categories, called dichotomies or binaryvariables • Two technical requirements for categories: • Exhaustive ( every observation fits into some category) • Leads to lots of “Others” • Mutually exclusive (every observation fits in exactly one category)

5. Levels of Measurement, cont. • Ordinal – same characteristics as nominal PLUS the fact that categories can be ranked from lower to higher • Mathematical operation of subtraction makes no sense, but > and < do • Most common: Likert • Interval – same characteristics as ordinal PLUS the fact that the arithmetic difference between any two values makes sense • That is, the usual subtraction operation makes the usual arithmetic sense • Ratio – same characteristics as interval PLUS the fact that there is a sensible zero value • Thus division and ratios make sense

6. Abbreviations often used for “Other” categories • NA – no answer or not answered • DK – don’t know • NAP – not applicable. Often this means the question was not even asked • nec or n.e.c.– not elsewhere classified. Typically in the category title, “Other, nec”

7. Ecological and Reductionist Fallacies • Unit of analysis – level (individual or some kind of aggregate) addressed by your theory or hypothesis • Unit of observation – level (individual or some kind of aggregate) from which data are collected • Ecological fallacy – drawing conclusions about individuals based on data from aggregates • Reductionist fallacy – drawing conclusions about aggregates based on data from individuals

8. Writing About Crosstabulations From a Sample • Lead with what is important • What’s important? • The fate of your hypotheses (if you have stated some) • The overall pattern for the dependent variable, especially if it is striking or surprising. Then look at deviations from the pattern in the categories of your independent variable • Big differences between categories of your independent variable • Things of interest to your audience • Remember, the usual point of a crosstab is to display differences between categories of the independent variable

9. Writing About Crosstabulations From a Sample, cont. • Do not use raw counts; use percents • Use the correct percents • Do not confuse row, column, and total percents • Be sure to specify the base for percents • Usually something like, “… x percent of [the base] …” or “Of all [bases] surveyed, x percent responded…” • Round percents in your text (but not necessarily in your tables) to integers • Be ready to convert percents to simple fractions • For example, 23 percent could be called “nearly a quarter” or “about one in four”

10. Writing About Crosstabulations From a Sample, cont. • Do not confuse percentage differences and percentage point differences • Percentage differences cannot be calculated by simple subtraction • Be ready to collapse categories • For example, to combine “Strongly agree” and “Agree” responses into one category • Be ready to calculate cumulative percents

11. Central Limit Theorem If repeated random samples of size n are drawn from any population with mean μ and standard deviation δ, the sampling distribution of sample means will be normal as n gets large, with mean μ and standard deviation δ/√n (also known as the standard error of the mean) . Hence, the standard deviation of the means drawn from many, many samples reflects 1) the standard deviation of the population, and 2) the sample size

12. Probability Sampling • Probability sampling is any method of drawing a sample of elements from a population such that the probability that any element or set of elements will be included in the sample is known and is not zero • The chief advantage of probability sampling is that the accuracy (or lack thereof) of estimates of population parameters from the sample can be estimated

13. Simple Random Sampling (SRS) • Frame – complete list of the survey population • Sample size – calculated based on desired precision of results • Selection rule – random selection without replacement • Estimate of population mean is the sample mean • Unbiased • s.e. = √fpc * (δ / √sample size)

14. Simple Random Sampling: Advantages and Disadvantages • SRS advantages • Samples are easy to draw • Samples are easy to use • Estimation of errors is “easy” • SRS disadvantages • Not always the lowest standard error method • Requires complete roster • Can be very expensive • Completing the frame may be expensive • Reaching geographically dispersed respondents may be expensive • May require large sample sizes to deal with rare population elements • Most elements in the sample will not be rare

15. Finite Populations and Sampling • Sampling error estimation depends on the Central Limit Theorem • The Central Limit Theorem applies to infinite populations • Infinite populations are easy to do in theory, but rare in practice • If you sample everyone in a finite population, the sampling error would be 0 • The closer you get to sampling everyone, the smaller your error should be • Central Limit Theorem says error is proportional to δ/√n

16. Finite Populations and Sampling, cont. • The finite population correction factor (fpc) takes into account the reduction in error you should get from sampling all or a large fraction of a finite population • The fraction of the population that is in the sample, n/N, is called the sampling ratio (f) • fpc = (N-n)/(N-1) ≈ (N-n)/N = (1 – f) • The standard error of the mean from a finite population (with simple random sampling) is√fpc * (δ/√n) • In practice, we ignore the fpc when the sampling ratio is less than 10%

17. Stratified Sampling • Frame • Usual SRS frame except broken into exhaustive, mutually exclusive groups • Requires knowledge ahead of time about how many elements in the population there are in each group • Each group is a stratum (plural strata) • Sample size - calculated based on desired precision of results • Calculations more complex than with SRS because there are more alternatives

18. Stratified Sampling (2) • Selection rules • Cases are drawn from each stratum • Cases within strata are drawn by SRS • Two alternatives for number drawn with each stratum • Proportionate to size – every element in the population has an equal chance of being drawn into the sample, regardless of stratum • Disproportionate – some strata will have a larger proportion of the sample than they will of the population

19. Stratified Sampling (3) • Estimation of the mean • If proportionate to size selection is used, the sample mean is an unbiased estimate of the population mean • If disproportionate selection is used, weights must be used to obtain an unbiased estimate of the population mean • Standard error of the mean will ordinarily be lower than the standard error from a simple random sample of the same size • The more homogeneous the elements are within strata, the more efficient stratified sampling will be

20. Stratified Sampling: Advantages and Disadvantages (compared with Simple Random Sampling) • Advantages • Reduced standard errors of estimate over SRS • Can thus get the same precision as SRS with smaller sample size • If proportionate selection is used, unweighted sample statistics can be used to estimate population parameters • Disproportionate selection can be used to get sufficient numbers of members of rare populations • Disadvantages • Requires advanced knowledge about stratum sizes • Disproportionate selection requires use of weights in making estimates of parameters

21. Cluster Sampling • Most complex method. Often used in conjunction with stratification and SRS; this is called multi-stage sampling • Frame • Broken into groups called clusters • Complete frame is needed only for clusters that are selected • It is necessary to know the size of clusters that are not selected • Sample size – usually calculated based on explicit tradeoff between costs and precision of results • Calculations more complex than with SRS or stratification because there are more alternatives

22. Cluster Sampling (2) • Selection rules • A sample of the clusters is drawn by simple random sampling • Within each cluster either all the elements or a simple random sample of the elements are drawn • When possible, sample sizes within clusters are drawn proportionate to size • NOTE that in cluster sampling only some of the clusters are used, while in stratified sampling, all of the strata are

23. Cluster Sampling (3) • Estimation of the mean • If clusters and elements within clusters were drawn so that all elements in the population had equal probabilities of selection, the sample mean is an unbiased estimate of the population mean. This rarely is possible • In the likely case of unequal probabilities of selection, weights must be used to obtain an unbiased estimate of the population mean • Standard error of the mean will ordinarily be higher than the standard error from a simple random sample of the same size • The more heterogeneous the elements are within strata, the more efficient cluster sampling will be • To the extent possible, each cluster should be representative of the entire population

24. Cluster Sampling:Advantages and Disadvantages (compared with Simple Random Sampling) • Advantages • Cost control • In general, the only reason to use clustering is to reduce financial or time costs • Can be used with stratification of clusters to help control standard errors • If proportionate selection is used, unweighted sample statistics can be used to estimate population parameters • Disadvantages • Sampling consultant probably needed • Larger standard errors than with SRS • Parameter and error estimation usually requires use of weights

25. Sample Pathologies • Biggest, most common problem: non-response • Estimation of parameters and errors assumes that data were collected from every element in the sample • Limitations on generalizability due to mismatch between the population of interest (target population) and the frame (survey population) • Called coverage error

26. Surveys á la Dillman:Eight Steps • Decide what information you need • Choose a survey method • Draw a sample • Write questions • Design the questionnaire • Field the survey • Turn answers into usable data • Report results Source: Patricia Salant and Don A. Dillman. 1994. How to Conduct Your Own Survey. NY: Wiley

27. Dillman on the Survey Process • Dillman analyzes the survey process from an exchange theory perspective • There is an exchange between the researcher and the respondent • Compliance with researcher’s request for information is a function of the social rewards the researcher can offer the respondent • Rewards such as gratitude, opportunity to have a say on something important

28. Writing Survey Questions • Question topics • There is little you can’t ask about • Useful distinction: • Questions about subjective states like attitudes, beliefs, and knowledge • Questions about objective phenomena like behavior or demographic attributes • Always remembering that in a questionnaire even objective phenomena are filtered through the respondent’s mind

29. Writing Survey Questions (2):Question Form • Two basic question forms: open-ended and closed-ended • Open-ended questions are questions to which respondents can give any answer • Closed-ended questions both ask a question and provide the respondent with preset answers to the question to choose among Pp. 177ff in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

30. Writing Survey Questions (3):Closed-ended Questions • Questions with ordered categories • E.g., Likert scale items • When there is an order, be sure to use it • Questions with unordered categories • Partially closed-ended • One option is something like“Other (please specify) ____”

31. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts • Avoid jargon, slang, and abbreviations • Avoid ambiguity, confusion, and vagueness • Whatever • Avoid emotional language • Can evoke frames that effectively hijack the intent of the question • Avoid prestige bias Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

32. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts (2) • Avoid double-barreled questions • Do not confuse beliefs with reality • Avoid leading questions • Avoid asking questions that are beyond respondents’ capabilities Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

33. Writing Survey Questions:Neuman’s Dirty Dozen Don’ts (3) • Avoid false premises • Avoid asking about intentions in the distant future • Avoid double negatives • Avoid overlapping or unbalanced response categories Pp. 170-3 in W.L. Neuman. 2007. Basics of Social Research. 2nd ed. Boston: Pearson

34. Questionnaire Layout (1) • Very important • Reflects your professionalism in the eyes or ears of your respondents and the eyes of your interviewers • Affects the likelihood of measurement error through respondent or interviewer error • Affects response rate • In mail surveys designed primarily with respondent in mind • In telephone and face-to-face surveys, designed with both interviewer and respondent in mind

35. Questionnaire Layout (2):Mail Surveys • Overall objectives • Minimize perceived (and real) respondent burden • Don’t confuse respondent • Simplify later data entry • Make a booklet • Questions are enclosed inside a booklet made of folded legal sized (8.5 x 14 inch) paper • No questions on the front or back of the booklet

36. Questionnaire Layout (3):Mail Surveys • Front page of booklet: • Title of study • Some graphic stuff • Sponsor • Return address • Back page • Request for comments • Thank you • Return address and telephone contact information

37. Questionnaire Layout (4):Mail Surveys • Overall question sequence • Start easy • First question must grab attention, reflect the issues in the cover letter, and not be too difficult or threatening • Start on topic • Group like questions together • Makes writing transitions easier • Keep threatening questions until later in the questionnaire • Get your demographics last • That’s probably least important to you and apparently least relevant to respondent

38. Questionnaire Layout (5):Mail Surveys • Layout of individual pages • Use white space • What counts is not how many pages the survey is, but rather how long it seems to be to respondents • Use fonts consistently to distinguish questions, answers, and instructions • Dillman likes to use bold for questions, all caps for answers, unbolded for transitions, and unbolded in parentheses for instructions • Establish a vertical flow • Precode the answers, usually on the left margin

39. Fielding Mail Surveys (1) Overview 1. We’re always trying to increase response rates 2. Respondents are most likely to respond if they think benefits outweigh their costs 3. We need to keep respondents engaged from the opening of the mail through the returning of the completed questionnaire Source: Salant and Dillman

40. Fielding Mail Surveys (2) Bottom lines 1. Mail survey response rates depend very much on the number of contacts 2. Mail surveys require advanced planning - Be sure you have the resources to meet the schedule 3. What really matters is the overall look and feel of the questionnaire - It’s a lot like buying (or selling!) a car Source: Salant and Dillman

41. Fielding Mail Surveys (3) • First mailout – advancednoticeletter • Sent to the entire sample • Mailed first class • Handwritten signature • Explains why there will be a survey • Explains why participation will be appreciated • Put yourself on the mailing list for this and all other mailings

42. Fielding Mail Surveys (4) • Second mailout – cover letter, questionnaire, and return envelope • Sent one week after advanced notice • Cover letter • Personalized • Explains survey purpose • Explains ID# on the questionnaire and promises confidentiality • Reinforces importance of everyone’s participation • Specifies who should complete the questionnaire • Thanks respondent for participation • Hand signed Source: Salant and Dillman

43. Fielding Mail Surveys (5) • Questionnaire – with ID number • Return envelope is stamped, addressed, and ready for use

44. Fielding Mail Surveys (6) • Third mailout – postcardfollowup • 4 to 8 days later • Personalized • Reminding and thanking • Fourth mailout – new cover letter, questionnaire, and return envelope • Three weeks after the second mailout (the first one with a copy of the questionnaire) • Sent only to addresses that have not yet returned the survey

45. Fielding Mail Surveys (7) • The four mailings should yield a final response rate of 50 – 60 percent • To further increase response rate, one can: • Send another follow up like the fourth mailing • Send the follow up as certified or express mail • Telephone • Often you will discover that people shouldn’t have been in the sample in the first place

46. Sampling Review • Rule of thumb sampling error of a proportion at the 95 percent confidence level = 1 / square root (sample size) • If size = 400, error = 1/20 = 5% • The Central Limit Theorem is important for social science research because it provides the mathematical basis for using probability samples 1) to make estimates of parameters from large populations using small samples and 2) to estimate the precision of those estimates

47. Sampling Review (2) • In both stratified and cluster sampling the survey population is divided into exhaustive, mutually exclusive groups. Each group could be either a stratum or a cluster • If we use all the groups in our final sample, we call each group a stratum • If we use only some of the groups in our final sample, we call each group a cluster