1 / 45

Survey Methods 2

Survey Methods 2. Class 20. Estimating Sampling Error Assume everything done correctly thus far: Specified sampling frame Stratified by area Sampled to accommodate underrepresented groups Is our survey error-proof? 100% reliable?

oded
Download Presentation

Survey Methods 2

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Survey Methods 2 Class 20

  2. Estimating Sampling Error Assume everything done correctly thus far: Specified sampling frame Stratified by area Sampled to accommodate underrepresented groups Is our survey error-proof? 100% reliable? No: Surveyed only 10% of population—sample may misrepresent source population. How accurate is our sample? Accuracy is a function of: 1) Sample size and 2) variability of thing measured Sampling error = amount sample differs from total population

  3. Standard Error of the Mean 1 SE = 67% of sample means 2 SE = 95% of sample means 2 SE = "Confidence interval" Standard Error = SE = Variance/n

  4. Standard Errors for Probability SE for Probabilities p = Probability that the characteristic exists 1 - p = probability that the characteristic does not exist p(1-p) = variance of proportion p(1-p)/n = SE of a proportion Example 30 % of FIASCO respondents say that arguments are related to money. Sample size (n) = 1500 SE = p(1-p)/n = .30 X .70 /1500 = .21/1500 = .01 Interpretation Confidence interval = 2 SE = 2 X .01 = .02 True mean (proportion) = .30  .02; or 95% confident that true mean lies between .28 and .32

  5. OOptimal Sample Size No Easy Answers—But a Possible Rule of Thumb  1. Samples of 150 fairly accurate: little gain in precision beyond n = 150 2. Consider the smallest sub-group you will sample—make sure that that group is sampled at n =150, as are all other groups. Then, make sure to weight groups according to their representativeness.

  6. Response Rate Computation Number Actually Sampled Number in Targeted Sample 1400 = 78% Response Rate 1800 Typical and Acceptable Ranges Office of Management and Budget (OMB) Standard: 75% Less than 30%: Self selected Common response rate: Between 25% and 90%

  7. Reasons for Non-Response Related to Bias Survey Procedure Respondent Interest Level Respondent Education Level Respondent Availability   Respondent Accessibility Respondent Ability to Participate Library Digest predicts Alf Landon for President, 1936

  8. Correcting for Non-Response Proxy Respondents OK for factual items (how often do arguments occur?) Not OK for subjective items (how do you feel about arguments?) Statistical Adjustments Re-weight sub-group with high level of non-response (But, what if non-respondents systematically differ from respondents?)  Survey Non-respondents (But, what if “responding” non-respondents systematically differ from “non-responding” non-respondents?)

  9. Developing the Survey Instrument

  10. Families in America, Survey of Conflicts (FIASCO) 1. Sampling frame * Nuclear families * S. Orange, NJ 2. Sampling strategy * Systematic stratified (strat. by quadrants) * Differential probability sampling (oversample new res.) 3. Contact strategy * Random digit dialing (RDD) * Only 1 person per home, parents/kids = chance 4. Sample size * n = 150 for smallest subgroup, based on est. of std. error 5. Non-response (NR) strategy * Means to monitor NR * Preventive measures (e.g., ease of response, follow-up calls) * Corrective measures (e.g., stat. adjustments, survey NRs)

  11. Survey Design 1. Define objectives 2. Draft and refine questions 3. Format questions 4. Field test instrument

  12. Defining Objectives 1. Statement of Purpose 2. Listing of main variables 3. Analysis Plan

  13. Statement of Purpose: FIASCO Survey The purpose of this survey is to assess family conflicts in S. Orange, NJ. Toward this end, the survey measures the type, frequency, and causes of conflicts in S. Orange families. The survey also attempts to relate family characteristics, and family member's attitudes, towards the type, frequency, and resolution of conflicts. Family characteristics -------> Conflicts Family attitudes -------> Conflicts

  14. Listing of Variables: FIASCO Study Independent Variables a. Family size b. Family composition c. Income level d. Education level e. ______________ Demographics a. When should conflicts be aired? b. How should conflicts be settled? Attitudes Dependent Variables a. Type of conflict b. Frequency of conflicts c. Causes of conflicts d. ______________ Conflicts

  15. Analysis Plan 1. Sampling Frame (e.g., stratified by income) 2. Sampling Method (e.g., RDD phone survey) 3. Type of inquiry: a. Associations (correlations)? b. Causes (experiment)? * Pre-post? * Between groups?

  16. Criteria for Good Survey Questions 1. Reliable Respondents understand question meaning Respondents understand what is expected of them 2. Valid Question item addresses underlying concept.

  17. Watz.’s INTERPRETATION:  Certain that he was being made the object of some incomprehensible but disrespectful joke. RECEPTIONIST'S INTERPRETATION:  First this guy says “my name is not Slavic”, then he says it is Slavic. He must be a new psychotic patient of Dr. H's. Road Bumps in Human Communication (Watzlawick, Beavin, & Jackson, 1967) Watzlawick (Vaht-slahv-ick) applied for an assistantship with a psychiatric research institute.  The following conversation took place with the receptionist. WATZ:  Good afternoon, I have an appointment with Dr. H. My name is Watzlawick RECEPTIONIST:  I didn't say it was. WATZ.:  (taken aback and somewhat annoyed).  But I'm telling you it is. RECEPTIONIST:(bewildered) Why then did you say it wasn't? WATZ.:  But I said it was.

  18. Specific Questions Must Map on to Subjects' Reality "Do people in your family employ a Socratic method of discourse when attempting to isolate the instigating cause of the discord under which you collectively toil?" "To what degree do people in your household argue about the influence of the International Monetary Fund (IMF)?"

  19. Directive Tell person to cheer up. Non-Directive Ask person how you can help. Tell person to cheer up Ask person how you can help. Don't let person stew in bad mood. Don't let person stew in bad mood. Listen without judging. Take charge of person's problem. Take charge of person's problem. Let person express any feelings. Listen without judging. Let person express any feelings. ???-Sorting

  20. Directive Tell person to cheer up. Non-Directive Ask person how you can help. Tell person to cheer up Ask person how you can help. Don't let person stew in bad mood. Don't let person stew in bad mood. Listen without judging. Take charge of person's problem. Take charge of person's problem. Let person express any feelings. Listen without judging. Let person express any feelings. Q-Sorting

  21. ??? Groups Best done with 6-8 people  Types of things participants can contribute: I. Understanding of questions a. Concepts b. Terms II. Insights related to topic a. Perceptions b. Experiences c. Feelings

  22. Focus Groups Best done with 6-8 people  Types of things participants can contribute: I. Understanding of questions a. Concepts b. Terms II. Insights related to topic a. Perceptions b. Experiences c. Feelings

  23. FIASCO Focus Group CONCEPTS AND WORDING "When do conflicts most often occur in your family?" "What do people in your family most often fight about?" ___ Finances ____ Chores ____ Noise ___ Messes ____ Other INSIGHTS What constitutes a conflict? What are the biggest problems with conflicts? How do conflicts get started? How do conflicts get settled? How do you feel about conflicts? WHAT'S MISSING?

  24. Wording Problems 1. Incomplete wording 2. Inappropriate optional wording 3. Poor word choice 4. Poorly defined terms 5. Special cases a. "Don't Know" b. Wording for special groups

  25. Incomplete Wording Poor form “Reason for last conflict?” A 1. Yes A 2. Because of competition for the TV remote. Better form "What was the main reason for the most recent conflict you personally had with someone in your family?" A 1. Because of competition for the TV remote.

  26. Unacceptable Optional Wording Proper optional wording: "You said your last conflict was with your [NAME FAMILY MEMBER] How often did you get into conflicts with (HIM/HER) during the last month?" Improper optional wording: "How intense was your last conflict" (Was there shouting, physical threats, acts of violence)?

  27. Unclear Meanings "On what calendared orientation point did the most recent conflagration transpire amongst those with whom you share primary genetic bonds?" PROBLEM? Overly Complex "Whom did you last see for professional advice regarding conflicts in your family?" "Are you punished if you don't complete routine chores (like housecleaning) on time"? PROBLEM? Ambiguous Terms

  28. Response Format for Open-Ended Questions "When do arguments most often occur in your household?" A 1: When mom’s tired A 2: On Sundays Problem? Specify times, quantities, frequencies "What things do you do to calm tensions between your siblings?" A 1: A zillion things. 1…., 2.….., 3.…., 4.….., 5.……, 100000000….. A 2: Take a deep breath Problem? Specify the number of points to be made. “Why do you and your father argue about money? A1: Because of rising tuition costs. A2: Because he’s a jerk. Problem? Focus the objective of questions.

  29. Open Ended and Closed Ended Questions Open Ended: Where respondent is free to provide an answer in his or her own words. "What topics lead to arguments in your family?" The things that lead to fracases in my family are usually about clothing. Mummy insists that we wear shirt and ties to the Sunday dinner table, but Natasha and Sterling just want to wear their day-glo capes and studded collars. Closed Ended: Where the respondent is restricted to a pre-set range of response options. "What topics lead to arguments in your family?" ___ Finances ___ Noise _X_ Decorum (manners, clothing, make-up) ___ Household chores ___ Opinions about social issues ___ Other

  30. Closed Ended vs. Open Ended Questions Closed Ended Defines question more clearly Defines response options more clearly Does not require an interviewer Easy for respondent Easy for coding/analysis Easy for data entry Open Ended Provides for unexpected responses Less likely to bias responses Enhances respondent good-will Historical record

  31. Not at all Slightly Somewhat Quite a Bit To a Great Degree Poor 1 2 Fair Good 3 Very Good 4 Excellent 5 1 2 3 4 5 Subjective Response Scales How would you rate the competence of your court-appointed family mediator? To what degree are family conflicts resolved in a way you find satisfactory?

  32. Considerations Regarding Questionnaire Item Format 1. Limited interpretability of subjective response scales 2. Agree-disagree options 3. "Don't Know" option 4. Middle position choice options 5. Unbalanced vs. balanced format 6. Measures of attitude strength

  33. Good Fair Poor How Do You Rate Robeson Cafeteria Food?

  34. Superb Very Good Good Fair Poor How Do You Rate Robeson Cafeteria Food?

  35. Agree/Disagree Format Agree Disagree • I wish my family would be more courteous. • I am tired of having to watch what I say. • Venting pent up feelings is selfish. • My family needs to see a conflict mediator. Problems with this format?

  36. Problems With Agree/Disagree Format • Loose info when squeezing range of attitudes into a dichotomy. Can’t correlate responses w’ each other. • Works best for extreme attitudes. Can’t distinguish moderate or ambivalent att’s from extreme att’s. • Confusing to indicate a negative (or positive) attitude by responding in the affirmative (or negative). Fowler discourages use of agree/disagree for these reasons.

  37. What Difference Does It Make To Include / Exclude DK (“Don’t Know”) Response? 1. Does it change rate at which people say DK? 2. Does it change the rate at which substantive (non-DK) responses are endorsed, relative to one another? 3. Does DK option change relationships between variables?

  38. Solution: Provide DK option.

  39. Solution: Provide DK option.

  40. “Floaters”: A Problem and a Mystery Floaters: People who ??? when there is not a DK (“don’t know”) filter, but who reply “???” when there is a DK filter. Relations between responses changes when DK filters are used. * Floaters ??? true correlations * Floaters can create ??? correls, where none actually exist * Floaters sometimes ??? Who are floaters????

  41. “Floaters”: A Problem and a Mystery Floaters: People who give a response when there is not a DK (“don’t know”) filter, but who reply “Don’t Know” when there is a DK filter. Relations between responses changes when DK filters are used. * Floaters weaken true correlations * Floaters can create false correls, where none actually exist * Floaters sometimes make no difference at all Who are floaters? Difficult to say. For obscure questions, floaters tend to be among the less educated.

More Related