Questionnaire Construction

1. Questionnaire Construction January 28 & 31, 2011

2. Objectives By the end of this meeting, participants should be able to: • Distinguish open and closed-ended questions. • Evaluate questions for their validity and reliability, as well as how well they follow basic rules of question wording.

3. Florida Datamar Democratic Polling Likely Democratic Voters Clinton 40% Obama 28% Edwards 19% Kucinich 2% Richardson 2%

4. The Vital Information- FL • Sample size 541 +/- 4.2 percent sampling error • January 5 – 7, 2008 • Methodology: Findings are from a January 5 – 7, 2008 survey using a comprehensive predictive model of “likely” voters, based on election cycles and other factors of voters in Florida. The survey was conducted using an automated telephone dialer and the voice of a professional announcer. • The targeted-registration-based-sampling (TRBS)* selection criteria were based on election cycles and other voter factors in Florida. Datamar proprietary algorithms were used to generate random samples from the target group for calling. • Where necessary, responses were weighted according to age, gender, and political party.

5. The Vital Information- FL Q18. If the primary election were held today, for whom would you vote? The candidates are: Joe Biden, U.S. Senator from Delaware; Chris Dodd, U.S. Senator from Connecticut; Bill Richardson, Governor of New Mexico; Mike Gravel, former U.S. Senator from Virginia; Dennis Kucinich, Congressman from Ohio; John Edwards, former U.S. Senator from North Carolina; Barack Obama, U.S. Senator from Illinois; Hillary Clinton, U.S. Senator from New York.

6. Research 2000 FL-D Democratic Primary Voters Clinton 50% Obama 28% Edwards 13% Others 4% Undecided 5%

7. Vital Information- Florida Research 2000 The Research 2000 Florida poll was conducted from January 14th through January 16th, 2008. There were a total of 500 likely Democratic and 500 Republican primary voters as well as 500 statewide primary voters interviewed by telephone. The margin of error is 4.5%. If the Democratic Primary for President were held today, which of the following candidates would you vote for? (Rotated)

8. Types of Questions • Open-ended questions • What do you like about the Democratic Party? • What do you like about the Republican Party? • Closed-ended questions • Multiple choice questions • Rating scale/ Feeling thermometer • Disadvantages/ Advantages of Each Approach

9. Rating Scales Common political types • Rating scales (generally seven or six points from very positive to very negative) • Thermometers (generally 1 to 100) • Semantic differential (very good-very bad) • Seven point scale (two concepts at each end)

10. Good Standard Practice • Use common questions • Use short questions with standard speech • Think of questions as a stimulus that the respondent will react to • Use a battery of questions for more difficult concepts • Tailor question wording to audience

11. Things to Consider • Branching format (particularly for scales) • Response choice order • Ambiguous wording • Biased questions • Social desirability bias • Double barreled questions • Double negatives • No opinion/ don’t know

12. Evaluating Questions • Reliability • Validity • Face validity • Convergent validity (similar answers) • Divergent validity (different answers) • Criterion validity (direct measure) • Content validity (all parts of a concept) • Construct validity (consistent with previous work)

13. Constructing a Questionnaire • Topic Order • Question order/ Consistency bias • Response choices/ Response set • Number of questions • half samples • rotating questions • Questionnaire layout • Pretesting the questionnaire • cognitive think-aloud

14. Example: Sensitive Questions • Suppose UGA wanted to survey student behavior on campus. • Specifically, the administration wants an accurate picture of what percent of students partake in unscrupulous activities. • How can the university ensure it gets accurate results?

15. Example: Sensitive Questions Innocuous Question Technique • Flip a coin. If you get heads, answer question 1. If you get tails, answer question 2. Do not tell me what you get or what question you answer. 1) I have had unprotected sex during the last month: Yes / No 2) I got tails (on the coin flip): Yes / No

16. Example: Sensitive Questions • Proportion of “Yes” responses =(Yes to question 1) + (Yes to question2)/Total # of respondents = • Yes to question 1/Total # of respondents + Yes to question2/ Total # of respondents= • θP + (1 − θ)δ, where θ=probability of answering question 1 P= proportion who had unprotected sex in the past month (1 − θ)= probability of responding to question 2 δ= the probability of getting tails (yes: Q.2)

17. Random Half Sample • Divide survey sample in half (randomly). • Half of respondents get column 1 the other half get column 2.

18. For January 31 • Read WKB chapter 4. • Answer the following questions: • Imagine we conducted a survey of 350 UGA students and found that 93% thought the Bulldogs would win the SEC championship. What is the margin of error (at 95%)? Based on that margin of error, we can be 95% sure that championship predictions lie in what range? • Imagine we conducted a survey of GA voters and found that Obama had approval of 55%. 750 people were surveyed, what is the margin of error (at 95%)? Based on that margin of error, we can be 95% sure that Obama’s support in GA lies in what range?

19. For February 2 • Read WKB chapter 5 • Answer questions 1, 2, & 3 on page 130.