Constructing the questionnaire

1. Constructing the questionnaire Survey Research and Design Spring 2006 Class #10 (Week 11)

2. Today’s objectives • To answer questions you have • To discuss analytic exercise #2 (and #3) • To understand questionnaire design and ways to construct questionnaire • To spend time applying information to several surveys Survey Resarch and Design (Umbach)

3. Primary objectives of design • Reduce nonresponse error and measurement error • Social exchange theory • Improve rewards by making it look interesting and important • Decrease costs by making it look easy to complete • Heuristics/satisficing • People tend to skim through surveys • Use simple rules of thumb to interpret what they see • End result: design is very important to a successful survey • Especially for mail and web surveys; they are self-administered Survey Resarch and Design (Umbach)

4. Paper survey format • You should avoid • Double sided paper with a staple in the corner • Landscape orientation • Unusual/multiple folds (like the day care survey) • Unusual shapes (e.g., square) • Use a booklet form instead • People are used to booklet forms; less likely to get confused, skip pages • Legal size paper or 11” x 17”, folded and stapled on spine • Use two-column format for 11” x 17” • Shorter text lines are better • If you must, print single-sided and staple in corner • Dillman says this looks less professional than a booklet Survey Resarch and Design (Umbach)

5. Web survey format • One page or multiple pages? • Depends if your software can handle multiple pages • Aldo how long your survey is – how long does it take to load when one page? • Avoid having too many separate pages; makes the survey seem long • Use some sort of progress indicator • Number of pages (1 of 4, 2 of 4) • Pie chart (20% complete, 40% complete) • Bar: • Make sure layout is vertical (reads from top to bottom) • Make sure the respondent does not have to scroll horizontally to read the survey questions Survey Resarch and Design (Umbach)

6. Beginning and end of survey • Beginning • For paper, Dillman discusses possible design options • Include general instructions and survey sponsor • For web, these can appear at top of the first page • End • Dillman recommends a thank you and a space for comments • Include contact information for questions • Web surveys: include instructions for submitting as well as a submit button Survey Resarch and Design (Umbach)

7. Question order • Avoid constantly switching topics • Try to sequence questions • By salience, with most salient at the beginning • Logically, e.g. by time • Group together questions that have similar components • Place sensitive and boring questions near the end of the survey • Less likely to object after filling out first part of survey • Examples: income, demographics Survey Resarch and Design (Umbach)

8. Question order effects • Norm of evenhandedness or value-based effect • Anchoring or cognitive-based effect • Addition or carryover effect and subtraction effect • Increased positive rating of summary items when asked after specific items on the same subject • More likely to occur with opinion question and vague quantifiers Survey Resarch and Design (Umbach)

9. Choosing the first question • Important because it gets the respondent “hooked” into survey • Four characteristics: • Easy to read and answer • Able to be answered by everyone • Interesting; should reference topic of survey described in contact • “Connectedness” Survey Resarch and Design (Umbach)

10. Define a navigational path for reading all information on each page • Minimize the need to reread portions in order to comprehend response task • Place instructions where needed and not at beginning of survey • Place items with same response categories in groups • Saves space because response category is not repeated • But be aware, respondents will view items as a group • Avoid combining questions to save space • In higher ed, students are often asked to rate themselves on an item, and also rate how important that item is, at the same time • Minimize use of matrices (Dillman p. 103) Survey Resarch and Design (Umbach)

11. Use visual elements as navigational guides • Increase font size to attract attention • Increase brightness or color to attract attention and establish groupings • Dillman bolds questions but not response categories • Use spacing and similarity to identify groupings • E.g, extra blank spaces between questions • Maintain consistent figure/ground format • Generally use black text on white or lightly shaded background • If shaded background, use white for answer spaces • Number questions and number them simply (1,2,3, etc.) Survey Resarch and Design (Umbach)

12. Use visual elements as navigational guides • When possible, place special instructions inside question numbers, don’t have them stand alone (Dillman p. 119) • Maintain simplicity, regularity, and symmetry • See Dillman pp. 111-112 Survey Resarch and Design (Umbach)

13. Visual design for questions • List answer categories vertically instead of horizontally; do not double- or triple-bank • Response categories should be evenly spaced • Important to check this for web surveys! • Response categories should be in correct order with conceptually neutral category in middle • Keep response scale direction consistent throughout survey • Separate at end of response category (with space or line) don’t know / not applicable responses • Think about size of text boxes for open-ended questions Survey Resarch and Design (Umbach)

14. Visual design for questions • Use yes/no questions rather than check all that apply Please check all that apply: Yes No □ Library Library □ □ □ Athletic facilities Athletic facilities □ □ □ Dining hall Dining hall □ □ □ Computer lab Computer lab □ □ • Use alternating shaded rows or columns for large batteries of items, or periods to connect response category to check-box Survey Resarch and Design (Umbach)

15. Skip instructions • These can be very problematic; require a lot of planning and testing • Directional arrows are key • Use a different type of directional arrow to indicate another path versus a skip • Increase font size of skip directions • Repeat qualifying answer in parentheses (e.g., “If yes”) • Indent the screened question Survey Resarch and Design (Umbach)

16. Review surveys • Paper • Survey of Earned Doctorates • http://www.nsf.gov/statistics/srvydoctorates/survey2004/sed_2004.pdf • National Survey of College Graduates • http://www.nsf.gov/statistics/srvygrads/survey2003/grads_2003.pdf • Your First College Year • http://www.gseis.ucla.edu/heri/yfcy/06%20PDFs/YFCY2006.PDF • Web • College Student Experiences Questionnaire (CSEQ) – not complete • http://www.cseq.org/ • Faculty Survey of Student Engagement (FSSE) • http://nsse.iub.edu/pdf/fsse/FSSE_2006.pdf • Spend approximately 5 minutes to review each survey for design aspects (not question wording or order) Survey Resarch and Design (Umbach)

17. For next class… • Readings: • Groves et al. – Chapters 9 & 11 • Dillman – Chapter 7 • AAPOR – Questions about IRBs • http://www.aapor.org/default.asp?page=survey_methods/IRBS_faq • NO CLASS NEXT WEEK. I know you’re disappointed.  Survey Resarch and Design (Umbach)