THE BASICS OF CRAFTING GOOD QUESTIONS

1. THE BASICS OF CRAFTING GOOD QUESTIONS Damon Burton University of Idaho

2. Smyth, Dillman & Christian (2007) studied how question construction influenced responses on the number of hours a day students studied. 3 types of response patterns were tested: A low range with 5 categories from .5 or less to 2.5 hours & a 6th category of more than 2.5 hours A high range consisting of 2.5 hours or less and 5 categories between 2.5 and 4.5 hours An open-ended question with no categories. Responses for students studying more than 2.5 hours varied by response categories, including: 30% for the low range, 71% for the high range and 58% for the open-ended responses. QUESTIONCONSTRUCTION CASE STUDY

3. For many students, estimating their average study time is a complicated task that requires averaging across weekdays and weekends and different times of the year. Respondents often look to the question and its accompanying response options for clues. Typically they assume that the range emphasized by the scale represents how many hours most students study. The middle option is often seen as representing the amount the average student studies. To save time, students may simply estimate whether they study more or less than an average student. RATIONALE FOR FINDINGS

4. What are the important issues to consider in crafting good survey questions?

5. What survey mode(s) will be used to ask the questions? Is this question being repeated from another survey, and/or will answers be compared to previously-collected data? Will respondents be willing and motivated to answer accurately? What type of information is the question trying to obtain? ISSUES IN CRAFTING SURVEY QUESTIONS

6. Different survey modes rely on different communication channels. Telephone interview respondents give and receive information through spoken words and their hearing system. Memory also becomes a more important factor. Web and mail questionnaires transmit information through the visual system. In mail and internet surveys, visual design elements become more critical. 1. SURVEY MODE USED

7. If a question was used in another survey, the objective typically is to replicate the item with minimal change. Government surveys and longitudinal studies often repeat items. In self-administered studies, not only should the question be the same but also visual design and layout of the item. Can items be modified to remain contemporary in wording and focus? 2. COMPARING TO PREVIOUS SURVEYS

8. Respondents may ignore instructions, read questions carelessly or provide incomplete answers. Assume the worst in designing items. Poor item design may reduce motivation. Items may be (a) difficult to read and understand, (b) instructions are hard to find, or (c) the response task is too vague. 3. ITEM MOTIVATES ACCURATE RESPONSE

9. Respondents are less likely to answer certain behavioral questions that they perceive as embarrassing or threatening such as sexual or criminal activity. Broad categories are more readily answered when dealing with financial information. For sensitive questions, wording and putting the question into context improve response rates. 3. ITEM MOTIVATES ACCURATE RESPONSE

10. Factual or demographic information is the easiest type of questions to answer (e.g., age). For attitude or opinion questions, elements of the item can influence the answering process, type of response requested, wording, and visual layout. 4. TYPE OF INFORMATION REQUESTED

11. Respondents may be influenced by the context of the question (e.g., solving environmental problems). For behaviors or events, if the item requires remembering too many details, respondents rely on survey design for answers. Memory fades over time, so asking questions about easily-recalled or memorable behaviors enhances recall (e.g., days walked to work this week or recent trip to Yellowstone). 4. TYPE OF INFORMATION REQUESTED

12. What factors should you consider in choosing an effective item format?

13. Multiple aspects of the wording and design of the question must work together to convey meaning. Choose appropriate question format(s). Understand the anatomy of a survey question and use that information to design effective items. Question design benefits from informed, multiple and diverse perspectives. A WHOLISTIC APPROACH TO QUESTION DESIGN

14. Open-ended questions are general and give respondents great freedom in answering (e.g., What are your greatest sources of enjoyment in basketball?). Closed-ended questions are used to force all respondents to make a choice from a limited range of options (e.g., multiple choice questions or rating agreement on a 5-point Likert scale. The choice of format needs to be based on the purpose of the question. A WHOLISTIC APPROACH TO QUESTION DESIGN

15. Strengths of open-ended formats allows respondents to freely answer without limitations, numerical responses can be reported exactly without vague labels or ranges, Weaknesses of open-ended formats more respondents skip this item format, biases arise more often, more variations in responses making it more difficult to analyze, and short answers more likely. OPEN-ENDED QUESTION FORMATS

16. Strengths of closed-ended formats can be analyzed statistically (i.e., ordinal or interval data), data results are produced quickly, numerical responses can be reported exactly without vague labels or ranges, Weaknesses of closed-ended formats ordering and grouping may have unintended effects on answers and make response harder, Responses are constrained by possible answers. Hard to identify new categories of information. CLOSED-ENDED QUESTION FORMATS

17. Nominal data provides an unorder set of response choices (i.e., can’t assume one response is higher or lower than another). Ordinal data has an order to the categories of response choices (i.e., good, average and poor) but can’t assume the size of the scaled magnitude between categories is the same or represents equal distance. Interval data has both order and equal intervals between response choices (i.e., 5 categories from very dissatisfied to very satisfied). TYPES OF DATA

18. The most important part of a survey question is the stem or the words forming the question. Additional instructions help respondents comprehend the meaning of questions or concepts. Crafting survey questions involves both choosing words and visually displaying questions and responses. ANATOMY OF A SURVEY QUESTION

19. What are the most important guidelines for creating good questions?

20. Make sure the question applies to all. Make sure the item is technically correct. Ask one question at a time. Use simple and familiar words. Select specific, concrete wording. Use as few words as possible. Use simple, complete sentences. Make sure “yes” means yes & “no” = no. Be sure the question specifies how to respond. GUIDELINES FOR CHOOSING WORDS & FORMING QUESTIONS

21. Questions for mail surveys sometimes try to reduce the number of questions each person has to answer, Avoid “skip” instructions, Avoid the word “if” which implies a response is not needed. Avoid providing only 2 bad options, especially if respondents have to answer. Provide a range of categories so every person has to respond to every question. 1. QUESTION APPLIES TO ALL

22. Use the right units of measurement so the question is easy to answer. For example, the question “How many feet tall is your horse?” is hard to answer because horses’ height is typically measured in “hands” (i.e., a hand = 4 inches). Credibibility and trust come with using the appropriate terminology for the audience and being technically correct. 2. ITEM TECHNICALLY CORRECT

23. Multiple topics in the same item make answering inaccurate. For example, the question “Do you subscribe to and read any magazines related to your job?” could be answered based on (a) subscribing, (b) reading or (c) both. Responses are quicker and more accurate when the item has a single topic. Credibibility and trust come keeping questions simple and easy to respond to. 3. ONE QUESTION AT A TIME

24. Keep readability at 5th grade or less and keep word length to 6-7 letters. Avoid formalized wording with too complex or technical wording. For example, use “tired” instead of “exhausted,” “work” instead of “employment,” and “correct” instead of “rectify.” Avoid abbreviations. Replace specialized terms and cliches. Credibility comes from simplicity. 4. USE SIMPLE, FAMILIAR WORDS

25. To get the response you want, you must be specific about the question you ask. For example, the question “How many times did you get together as a family last week?” is somewhat vague. What does “get together as a family mean?” “How many meals did you sit down to eat as a family last week?” is much more specific. Specificity in questions allows much more accurate responses. 5. USE SPECIFIC WORDING

26. Keeping readability low usually reduces the number of words per item. Replace complex words or rewrite item more simply. Eliminate redundant words. However, keeping readability low and content specific is more important than question length. 6. USE FEW WORDS PER ITEM

27. Eliminate phrases or clauses. Write complete sentences that use a simple sentence structure. Avoid convoluted sentence structure with confusing wording. Keeping readability low requires simple sentences that are easily understood. Compound sentences work better than complex ones. 7. USE COMPLETE SENTENCES

28. The mental connection for the word “not” is difficult to comprehend. Specify meaning of words such as “favor” and “oppose” so “for” and “against” may be better terms. Ballet initiatives are often worded in a confusing ways so you are not sure whether you are “for” or “against.” Avoid this problem in your surveys. Credibility and trust come from making decisions easy to understand. 8. MAKE SURE “YES” MEANS “YES”

29. Match the question stem and the response options. Make the task clear to the respondent after have read the question once. The question stem has to clearly state the response task. The response format and/or options provided must match the task as stated in the question stem. Mismatched question and response makes item unusable. 9. SPECIFY THE RESPONSE TASK

30. What design elements should be considered in presentation of survey questions?

31. Visual layout influences how respondents organize information and focus attention on responses. 4 types of visual design elements communicate meaning and can be manipulated to draw attention to or take away from meaning. Words – the primary source of meaning. Numbers – convey meaning and sequence. Symbols – figures that add special meaning. Graphics – simple and complex shapes and visual images that convey meaning. VISUAL PRESENTATION OF SURVEY QUESTIONS

32. Visual design an be modified through font size and type, brightness and location of elements. Gestalt principles of pattern perception assign meaning and provide a sense of grouping and separation of elements. Proximity – locating elements together denotes relatedness. Common Region – boxes and highlighted areas designate connectedness. Connections – linking elements thru lines. Continuity – multiple elements layered continuously creates a perception of a group. VISUAL PRESENTATION OF SURVEY QUESTIONS

33. What are the primary guidelines for using visual elements to design questions and the overall survey?

34. Use darker and/or larger print for the question and lighter and/or smaller text for answer choices and spaces. Creates clear subgrouping and separation, Contrast helps delineate questions from answers. Use spacing to help create subgrouping within a question because proximity states that items located together are perceived as a group. Visually standardize all response spaces. Similarity states that items appear regular and similar are perceived as belonging together. VISUAL PRESENTATION GUIDELINES

35. 4. Use design properties to emphasize elements that are important to the respondent or de-emphasize ones that are unimportant. Manipulating size, contrast and location will emphasize or de-emphasize element importance. 5. Use design properties with consistency. Use each design element for only one purpose such as underlining to draw attention to important words or bolding for questions. 6. Visual elements in question must send consistent messages. Avoid conflicting messages communicated by incongruous visual elements. VISUAL PRESENTATION GUIDELINES

36. 7. Integrate special instructions into the question where they will be used. When special instructions are outside the question stem, they may be ignored. 8. Separate optional or occasionally needed instructions from the question stem by font or symbol variation. Distinguish between words that are essential for everyone to read and those that may be needed by only some respondents. 9. Organize questions in ways that minimize the need to reread portions of instructions in order to comprehend the response task. Emphasize respondent efficiency so they are always clear about what to do. VISUAL GUIDELINES

37. 10. Choose line spacing, font and text size to ensure legibility. Need appropriate font so avoid script and use serif or sans serif. Use 10-12 point font for most populations and larger fonts for older populations. Line length of 3-5 inches is recommended. VISUAL PRESENTATION GUIDELINES

38. End END