Using Surveys to Maximize Strategic Planning

1. Using Surveys to Maximize Strategic Planning CAUCE Dial and Discuss Tuesday, 23 February 2010 Stanley VarnhagenUniversity of Alberta Karen HaywardUniversity of Saskatchewan

2. Overview • Surveys and the Planning Process • Understanding Surveys • Delivery Methods • Closing Tips Dial and Discuss: Using Surveys

3. Surveys and the Planning Process Dial and Discuss: Using Surveys

4. Surveys and Planning Dial and Discuss: Using Surveys • Are surveys needed/useful for planning or do they just get in the way (slow things down)? • Depends on the timing of the planning • Useful for evidence-based decisions • Surveys can be an efficient means of collecting data • Properly done, they can be a valuable tool that increases the chances of success

5. Surveys are useful when… Dial and Discuss: Using Surveys information is needed to examine population characteristics a larger response pool is needed examining subgroup data is important (e.g., gender or age differentials) breadth of response is more important than depth

6. Surveys & Evaluations Dial and Discuss: Using Surveys • Surveys are effective tools with most types of evaluations including: • Needs assessments • Collecting information about the target audience • Formative evaluations (midterm) • Collecting information that helps make adjustments & improvements • Summative evaluations • Collecting information that helps determine if goals and objectives are being met (merit, value, & worth) • Maintenance evaluations • Collecting information a period of time after whatis being evaluated is complete, to see if the objectives are still being met

7. Surveys and Strategic Planning Dial and Discuss: Using Surveys Surveys can help us: Better understand potential audiences/users/students (consider different segments) Collect baseline data that can be compared with subsequent survey data Test assumptions or preliminary products (trial balloons) Examine existing and potential markets

8. Understanding Surveys… Dial and Discuss: Using Surveys

9. Collect suggestions to improve Catch problems Track trends Get anonymous feedback Get buy-in Gain understanding Increase awareness 2 way process Easy and effective way to collect data With an appropriate sample and response rate quantitative data can help generalize to the population What Surveys Can Offer Dial and Discuss: Using Surveys

10. Strengths of Surveys • Easy to create • Easy to analyze • Relatively inexpensive • Common/respondent familiarity • Handle both quantitative and qualitative data • Can be relatively time efficient to take • Real time results • Instant Feedback Dial and Discuss: Using Surveys

11. Additional Reasons to Use Surveys Dial and Discuss: Using Surveys • Can be cost savings, to allow feedback before considerable time and resources spent • Storyboards/scripts • Allow feedback while modification is still possible • Helpful to the developer and the user • Determine if goals and objectives have been met

12. Weaknesses of Surveys • Lack of follow-up • Not best suited for collecting qualitative data • Limited flexibility • Sampling problems not always obvious • Can miss key results • Results are dependent on the questions/options that are presented • Garbage In Garbage Out (GIGO) Dial and Discuss: Using Surveys

13. Delivery Methods Dial and Discuss: Using Surveys

14. Survey Delivery Methods Dial and Discuss: Using Surveys • Surveys can be conducted using different delivery methods • These methods have different strengths and weaknesses that should be considered • The three different delivery methods considered here are: • Online • Paper-based • Telephone • Currently Online Surveys are probably themost popular method

15. Advantages: Minimal distribution costs Automatic data entry Automatic compilation possible Branching possible Potentially quickly modified or updated Real time data collection potential Can use multimedia Disadvantages: SPAM Filters/Broken Links Duplication a concern User competency dependent (submit) More complicated Anonymity a concern Potential user cost (Long distance/ISP) Access Online Dial and Discuss: Using Surveys

16. Other Online Considerations • Response rate • Length • Item types • Forced choices • Making changes • Navigation • Going to previous pages • Submitting survey • Accommodating schedules • Different browsers • Colors • Size • Exactly who is responding • Location of the survey server • Online survey fatigue Dial and Discuss: Using Surveys

17. Advantages: Simple (for respondents and evaluator) No special training required Easy to go back and change responses Can have a higher response rate Disadvantages: Distribution Can be slow Branching Following instructions Usually required manual data entry Reading writing Qualitative responses Costs Paper and Pencil Dial and Discuss: Using Surveys

18. Advantages Can sometimes provides higher response rates Ability to branch Limited ability to follow up response Allows assessment of response quality Longer surveys possible Disadvantages Cost Interviewer Long distance Persistence More obtrusive Can be perceived anonymity problems Selective responses Caller ID & Voicemail Telemarketers Phone Dial and Discuss: Using Surveys

19. General Survey Considerations • Use (needs assessment, formative, or summative) • Data entry method • Number of response options • 5, 7, 99, or unlimited • Even or odd • Effective scale • Anchors • Continuum versus descriptors • Statement questions • Normative referenced • Validity • Reliability Dial and Discuss: Using Surveys

20. Other Considerations • Completion Time • Written versus telephone • Class versus own time • Mixed methods • Triangulation • Piloting • Incentives • Routing • Funding • Bias • Terminology • Acronyms • Reading ability Dial and Discuss: Using Surveys

21. Other Considerations (cont’d) • Anonymity • Aggregating data • Removing identifiers • Looking at trends across surveys • Differential responses • Only the “keeners” complete or vice-versa • Some other important characteristic determines who responds • Sampling • Social desirability • Time survey is available Dial and Discuss: Using Surveys

22. Bias Dial and Discuss: Using Surveys • Can affect results, but is often not obvious • Question wording • Leading questions • Options available • Missing options • Question order • Specific to general • Carry over • Option order • General survey expectations

23. Creating Surveys… Examples Dial and Discuss: Using Surveys

24. Item Types • Likert - Interval data • how many options • even or odd • additional options • direction • anchors I feel very comfortable taking online distance courses. 1 2 3 4 5 Strongly Disagree Neutral Agree Strongly disagree agree Dial and Discuss: Using Surveys

25. Uses Likert Items Five options Standardized Normative Effective scale Social desirability How would we know when assumptions have been violated? Intuitive meaning End-of-course Evaluations Dial and Discuss: Using Surveys

26. Item Types (continued) • Modified Likert • Not agree/disagree items • Continuum (options) changes • Can use fewer anchors • 2, 3, or 5 How comfortable do you feel online distance courses? 1 2 3 4 5 Not at allSomewhatVery comfortablecomfortablecomfortable Dial and Discuss: Using Surveys

27. Multiple choice Nominal data Reporting results Picking the correct choice Instructions Select best Select all that apply Missing options Where do you plan to complete most of your online coursework? a) Home b)Work c) Computer labs d)Not sure e) Other (please specify): Item Types (continued) Dial and Discuss: Using Surveys

28. Item Types (continued) • Rankings • Following instructions • Can force distinctions • Can be harder to interpret results • Fill-in-the-blanks • Can be time consuming to code • Generalization • Open ended • Use open-ended questions carefully • Exploratory versus confirmatory • Time • Potentially rich Dial and Discuss: Using Surveys

29. Closing Tips Dial and Discuss: Using Surveys

30. Survey Writing Tips • Demographic information • Focus on what information will be useful for analysis • Recognize many consider the information given to be sensitive • Caution about subject identification • Trade-offs • Don’t make responses too easy Dial and Discuss: Using Surveys

31. Survey Tips (cont’d) • Instructions • Clarify purpose/reasons • Anonymity • Voluntary • Withdrawal • Ethics • Implied Consent • Dissemination • When and where results will be found Dial and Discuss: Using Surveys

32. Survey Pitfalls • Want to get the largest proportion possible to respond • Reduced the chance of self-selection • Increases likelihood of accurately generalizing results • Increasingly difficult to get high response rates • Frequency being surveyed • Survey fatigue • Perceived need/relevance Dial and Discuss: Using Surveys

33. Tips to Increase Response Rates Dial and Discuss: Using Surveys • Well written cover letter from meaningful individual • Explaining survey purpose and relevance • Honorarium – given with survey • Draw (e.g., gift certificate) • Keep survey as short as possible • Provide reasonable deadline • Reminders

34. Reference Dillman, D. A., Smyth, J. D., & Christian, L. M. (2009) Internet, Mail, and Mixed-Mode Surveys: The Tailored Design Method (Third Edition). New York: John Wiley & Son. Dial and Discuss: Using Surveys

35. Questions? Dial and Discuss: Using Surveys

36. Contacting Us • Stanley Varnhagen • University of Alberta • phone: (780) 492-3641 • email: stanley.varnhagen@ualberta.ca • Karen Hayward • University of Saskatchewan • phone: (306)966-7384 • email: Karen.Hayward@usask.ca Dial and Discuss: Using Surveys