Advanced survey design

1. Tuesday, February 7, 2019 10:00 – 11:30am Advanced survey design Jen Sweet Center for Teaching & Learning DePaul University Shannon Milligan Institutional Research & Market Analytics DePaul University

2. Workshop Outline • Introductions • Conflicting Recommendations for Survey Design? • Cognition and Survey Design • Recommendations to Reduce Cognitive Load ACTIVITY! • Affective Assessment and Surveys ACTIVITY! • Survey Design & Distribution Tools

3. Workshop Outcomes By the end of this workshop, participants will be able to: • Apply knowledge of the cognitive processes students use to respond to surveys to design effective survey items and instruments. • Identify non-cognitive variables that can be assessed with surveys • Use the tools available to them at their respective institutions to design and distribute surveys.

4. Conflicting Recommendations for Survey Design?

5. Recommendations for Survey Design that seem to Conflict Examples: Neutral Point • Always include? • Never use? • Sometimes yes; sometimes no? Number of Scale Points to Include • 2? • 3? • 4? • 5? • 6? • 7? • 9? • The more the better? All of This is in the Literature!

6. So, What’s Up with the Literature? All of these recommendations may be appropriate depending on the specific context: • respondent attributes • nature of the items in the survey • length of the survey • Etc. Generally looking at things like: • Reliability • *Validity • Survey Outcomes • Response Rate (high) • Use of Response Sets (low) Survey Design is as Much Art as Science! “There is always a well-known solution to every human problem - neat, plausible, and wrong.” H.L. Hencken

7. Cognition and Survey Design

8. Cognitive Load Paas and Van Merrienboer, 1994 The amount of cognitive effort (or thinking) students need to exert to respond to a survey item. If cognitive load exceeds the student’s working memory capacity, they will take some sort of shortcut (Paas & Van Merrienboer, 1994), or satisfice (Krosnick, 1991) • Read questions less carefully (skim) • Use a response set • Give same response for all questions, regardless of content • Overuse neutral or N/A response option • Skip the question (provide no response) • Respond randomly • Decide not to complete the survey

9. Cognitive Steps to Respond to Surveys (Tourangeau, 1984) • Interpretation • Retrieval • Judgment • Response

10. Recommendations to Reduce Cognitive Load

11. Step 1: Interpretation • Use language that is clear and familiar to survey respondents. • Avoid cognitively taxing wording. • Avoid unfamiliar words and phrasing. • Avoid jargonand acronyms • Ensure that question stems are clear and explicit. • Do not use concepts that are unclear or unfamiliar to respondents. • Avoid complex sentence structures. • Ask about only one concept in each stem; avoid double-barreledquestions • Use questions that do not make assumptions. • Ask for information in a direct manner by avoiding double negatives • Ensure question stems are succinct, including only as much information as is necessary for respondents to properly interpret what is being requested of them.

12. Interpretation (Continued) • Include clear instructions that clarify the purpose of the survey instrument, and provide respondents with expected procedures for responding to the survey instrument. • Ensure that every portion of a survey instrument is visible without the need for additional action by the respondent. • Use radio buttonsinstead of drop-down boxesto display response options. • Do not “hide” definitions respondents may need to interpret and respond to survey items. • Use easy-to-read font size and type. • Use high-contrast font and background colors.

13. Step 2: Retrieval • Use stems that request information with which respondents have primary experience and avoid asking for second-hand information (i.e., information that the respondent has heard about, but not experienced personally) or hypothetical information. • Group conceptually similar items together.

14. Step 3: Judgment • Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about. • General Guideline: four or five response options, depending on whether or not there will be a neutral option. • Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided • Neutral responses can be difficult to interpret • Offering a neutral option may encourage satisficing

15. Step 4: Response • Use the smallest number of response options necessary to encompass all meaningful divisions of what you are asking about. • Label the scale options. • May only need to label the most extreme options • Include a neutral option if you reasonably expect participants to have no opinion, but otherwise, they should be avoided • Neutral responses can be difficult to interpret • Offering a neutral option may encourage satisficing

16. Activity!

17. Practice Evaluating Survey Items! Individually: Complete the worksheet In Groups: Compare Responses Did Everyone Identify the same items for Improvement? Are there differences in ways you edited items?

18. Surveys and the Rise of Affective Assessment

19. Growing Emphasis on Non-Cognitive Abilities • Grit • Growth • Social-emotional development • Self-awareness/management/efficacy • General affect • Engagement • Mattering • Climate • Research shows strong relationships between these variables and overall success From NPR: “Nonacademic Skills Are Key to Success. But What Should We Call Them?” (May 28, 2015)

20. The Role of Surveys From NPR: “To Measure What Tests Can’t, Some Schools Turn to Surveys” (December 2, 2015) From that article: “A growing battery of school leaders, researchers and policymakers think surveys are the best tool available right now to measure important social and emotional goals for schools and students” Why? • Easy to administer • Easier to collect and analyze than reflection papers (and the like) • Many surveys/survey questions already exist • Faster data sharing = faster decision-making/implementation (maybe)

21. Activity!

22. Non-Cognitive Assessment and You Individually: • What non-cognitive variables might be of interest to your program? • Are you already assessing any of these variables? In Groups: • Share what you are assessing and what you might be interested in assessing • What variables might be important to look at on an institutional-level?

23. Survey Design/Distribution Tools

24. Three Main Tools Used at DePaul • Qualtrics • Google Forms • Survey Monkey

25. Qualtrics Advantages: • Free to DePaul faculty and staff • Supported by Information Services • Very Flexible and Comprehensive System • Lots of Features Disadvantage: • Reporting Features aren’t Great • Steeper Learning Curve than Other Systems

26. Google Forms Advantages: • Free to Everyone • Data is Collected in Excel Format • Easier to Learn than Qualtrics Disadvantage: • Not Nearly as many Features as Qualtrics

27. Survey Monkey Advantages: • ? Disadvantage: • Most Advanced Features Cost $ • To get all the features available in Qualtrics, cost is $780/year • Limited to 10 questions and 100 responses on the free version

28. Questions??

29. Contact Information Jen Sweet DePaul University Office for Teaching, Learning & Assessment jsweet2@depaul.edu Shannon Milligan DePaul University Institutional Research & Market Analytics shannon.milligan@depaul.edu