1 / 46

Agenda

Agenda. Question exercise Question asking Issues and problems Types of questions and examples Advice for questionnaire construction. Question Exercise. Hypothesis: Exposure to sex content on television leads to risky sexual behavior Design two questions to address this hypothesis.

kasi
Download Presentation

Agenda

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Agenda • Question exercise • Question asking • Issues and problems • Types of questions and examples • Advice for questionnaire construction

  2. Question Exercise • Hypothesis: Exposure to sex content on television leads to risky sexual behavior • Design two questions to address this hypothesis

  3. Question asking • Interviewing is a social transaction • Problems: • Memory • Motivation • Communication • Knowledge

  4. Memory • Particularly problematic with low-salience behaviors • Aspects of the problem: • Failed recall • Misremembering • Telescoping • Cognitive priming

  5. Memory • Design considerations: • Specific questions • Detailed behaviors • Specified time periods • Aided-recall procedures • Bounded recall • Longer questions with memory cues • Diaries • Managing question order and context

  6. Motivation • Aspects of the problem: • Low motivation • Inattention • Acquiescence • “Satisficing” • Self presentation • Avoidance of threat • Social desirability

  7. Motivation • Design considerations: • Reduce respondent burden • Vary response formats • Legitimize “undesirable” responses • Work up to difficult questions • Reduce salience of threatening questions • Increase anonymity • Assess perceived threat

  8. Communication • All interviewing involves social interaction • Aspects of the problem: • Semantics • Multiple opportunities for misunderstanding • Conversational norms • Many implied meanings and assumptions • Respondents interpret purposes

  9. Communication • Design considerations: • Everyday language and concepts • Avoid specialized terms • Avoid if possible rates, proportions, numbers • Single clear object per question • Short question series • Avoid double negatives (watch out for not) • Appropriate response options (may suggest distribution of answers) • Manage question order and context

  10. Knowledge • Aspects of the problem: • Weak knowledge base • Minimal opportunities to inform

  11. I’d like to describe to you a new kind of insurance plan that would minimize the risk involved in investing in the stock market. You might want to read along with me on this card. (HAND RESPONDENT CARD) According to this plan, you would be free to choose any stock like U.S. Steel, AT& T, or General Motors. An absolutely reliable company would guarantee to buy back the stock from you after ten years, but not earlier, for what you paid plus any gains in the value of the stock since you bought it. If the value of the stock decreased during the ten years you would still get back your original investment and the company, not you, would bear the loss. The price you would pay for this insurance loss would be the dividends on the stock. This means that you would give up any dividends the stock would pay during the ten-year period in exchange for this guarantee against loss. If you wished to leave this insurance plan before the ten years were up, you would be free to do so and could sell the stock for the current market price. If the stock had increased in value, you would profit. If it declined, you would bear the loss, if you left the plan before the ten years were up. For leaving the plan, however, you would have to pay some charge like one or two years of dividends, which would vary according to how long you’d been in the plan. Remember, the firm offering this plan is absolutely reliable and the plan would be guaranteed by a major reputable insurance company. If such a plan were available, under which you would give up dividends on your stock in exchange for insurance against any loss on the money you put in, how interested would you be in participating in it -- would you be definitely interested, do you think you might be interested but would like more information, do you think you probably would not be interested but would like more information, or would you definitely not be interested at all? Harris (1974), cited in Converse and Presser (1986)

  12. Knowledge • Design considerations: • Simple language • Forsake rapid absorption of complex information • Explicit “don’t know” and “no opinion” options • Filters for some opinion items • Beware hypothetical questions

  13. Types of Questions • Behavioral Questions • Questions about Attitudes and Intentions • Open Ended versus Closed Questions • Knowledge Questions • Questions Evaluating Performance • Psychographic Questions • Standard Demographic Questions

  14. Types of Questions • Behavioral Questions • Non-threatening • Threatening

  15. Types of Questions • Non-threatening Behavioral Questions • May be individual-specific • Social changes • Social desirability • Is there a right or wrong answer?

  16. Types of Questions • Non-threatening Behavioral Questions • How to tell? • Previous questions • Pre-test

  17. Types of Questions • Non-threatening Behavioral Questions • Physical exercise (Group 2) • NHIS (2008)

  18. Types of Questions • Non-threatening Behavioral Questions • Physical exercise (Group 2) • Q1: How often do you do VIGOROUS leisure-time physical activities for AT LEAST 10 MINUTES that cause HEAVY sweating or LARGE increases in breathing or heart rate? • Q2: How many times per (day/week/month/year) do you do these activities?

  19. Types of Questions • Ways to improve Non-threatening Behavioral Questions • Aided recall • Specific wording • Specific behavior • Bounded recall • Secondary research • Diaries and panel research • Appropriate lengths

  20. Types of Questions • Threatening Behavioral Questions • Socially desirable versus undesirable • Approaches (Barton, 1958) • Causal • Numbered options • “Everybody does it” • “Other people have done it” • Sealed ballot • Kinsey technique

  21. Types of Questions • Threatening Behavioral Questions • Example from Anti-Drug Campaign Evaluation • The next series of questions ask about cigarettes, alcohol and various drugs. There is a lot of talk these days about these subjects. Remember that your answers will be confidential. We hope you will answer all of our questions. However, if there are some that you do not want to answer, you may skip them.

  22. Types of Questions • Threatening Behavioral Questions • Example from Anti-Drug Campaign Evaluation • Q1: Have you ever smoked part or all of a cigarette? • Q2: How old were you the first time you smoked part or all of a cigarette?

  23. Types of Questions • Threatening Behavioral Questions • Example from Anti-Drug Campaign Evaluation • We want to learn more about what young people think are the good and bad things that might happen if they use marijuana. We also want to learn more about how they and the people they know feel about using marijuana. The next series of questions is about marijuana.

  24. Types of Questions • Threatening Behavioral Questions • Example from Anti-Drug Campaign Evaluation • Q1: Who, if anyone, has offered you marijuana? Choose all that apply. • No one has ever offered me marijuana/My brother-sister-stepbrother-stepsister/My friends/ Other kids/Adults I know/Other adults

  25. Types of Questions • Ways to improve Threatening Behavioral Questions • Self-administered methods • Card sorting and randomized response • Open-ended questions • Familiar wording and long length • Diaries and panels • Informants

  26. Types of Questions • Questions about Attitudes • Identify object of attitude • Identify target component • Cognitive • Evaluative • Behavioral • Assess strength

  27. Types of Questions • Questions about Attitudes • Example from ANHCS and Bell et al (1999) • Q1: Please indicate the extent to which you agree or disagree with the following statements:

  28. Types of Questions

  29. Types of Questions • Questions about Attitudes • Argument Repertoire (Group 3) • Cappella, Price & Nir (2002)

  30. Types of Questions Q2: What are the reasons you have for feeling (very/somewhat) favorable toward the Democratic party? (Please list all the reasons that come to mind) Q3: What reasons do you think other people might have for feeling unfavorable toward the Democratic party? (Please list all the reasons that come to mind)

  31. Types of Questions • Questions about behavioral intention • Likelihood versus incidence • Frequency and future behavior • Unipolar versus bipolar • Multidimesionality • Using filters • Double-barreled and 1 1/2 barreled • Position • Unintentional activation of norms/values

  32. Types of Questions • Questions about behavioral intention • Example from Anti-Drug Campaign Evaluation • We want to learn more about what young people think are the good and bad things that might happen if they use marijuana. We also want to learn more about how they and the people they know feel about using marijuana. The next series of questions is about marijuana.

  33. Types of Questions • Questions about behavioral intention • Example from Anti-Drug Campaign Evaluation • Q1: How likely is it that you will use marijuana even once or twice over the next 12 months? (When we say marijuana we mean marijuana or hashish). • Definitely not/Probably not/Probably will/Definitely will/Refused/Don’t know

  34. Types of Questions • Psychographic Questions • Conformity (Group 1) • Feldman (2003)

  35. Types of Questions • Psychographic Questions

  36. Types of Questions • Psychographic Questions • Body of literature • Feedback from relevent experts in the area • In-depth interviews/focus groups • AIO statements measured on scales • Important behaviors included • Pilot study and multivariate analysis

  37. Questions to ask yourself • Is the question necessary? • Do I need several questions on this topic? • Do respondents know enough to respond? • Is the question appropriately specific and concrete? • Is the question understandable? Unbiased? • Will respondents answer it accurately?

  38. Steps to follow • Consult others • Other surveys, experts, insiders • Pretest everything • Usually multiple steps • Compare close-ended to open-ended responses • Begin with longer instrument (< 2 times longer) • Sample (e.g., 25-75) target population • Observe interviews

  39. Steps to follow • Evaluate pretests • Respondent understandings (don’t solicit advice) • Question results (means, variances, distributions) • Task difficulty • Respondent interest • Interviewer experiences • Flow / naturalness of interview • Question order • Skip and branch patterns

  40. Steps to follow • Use Second Pretest • “Full dress rehearsal” • Consider: • Random split-sample comparisons • Open-ended follow ups to closed questions • Random probes • Some follow ups for each respondent, spread randomly over the questionnaire

  41. Other concerns • Visual design key in mail, Web surveys • Clear instructions • Uncrowded displays • Natural reading patterns • Careful skip and branch design • Visual markers to direct attention

  42. Other concerns • Visual design key in mail, Web surveys • Example from SSB-CECCR Project

  43. General advice = Validity The goals: Precise and accurate responses = Reliability • General advice: Avoid reinventing the wheel • Draw upon empirical evidence • Consider existing questions and formats • Pretest, and pretest again

More Related