21 tips for conducting evaluation research on a boot lace

1. David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation research on a boot lace

2. Today’s objectives • Advice on doing it yourself • Advice on being a good client

3. Where to get help • University departments • Social work, Sociology, Psychology, Political Science, Business, and others • Contacts via board of directors • American Marketing Association • Market Research Association • Market research firms • LinkedIn (www.linkedin.com) • National Opinion Research Center (University of Chicago) http://www.whatisasurvey.info/

4. http://www.italladdsup.gov/community_partners/dc_teachmehow_02.asphttp://www.italladdsup.gov/community_partners/dc_teachmehow_02.asp • Introduction to evaluation research • Process evaluation examples • Telephone survey with annotations • Intercept and self-administered survey

5. Build a community • Vision • Lutheran Social Services Research Community • Share best practices • Methods • Facebook or other collaboration tool

6. 1 Set your objectives • Program objectives • Specific information needs

7. 2 Do your background research

8. 3 Think about different types of evaluation and measurement • Formative evaluation • Helps planning • Process evaluation • How well is the program operating? • Impact evaluation • Building awareness and understanding? • Outcome evaluation • Concrete program or “business” metrics

9. 4 Don’t assume you know audience attitudes and opinions • You need formative research

10. 5 Set up your process metrics

11. 6 Measure impacts • Awareness • Knowledge • Attitudes and perceptions • Intended actions

12. Known respondents Population =Sample frame Members Donors Clients Employees Civic organization leaders Defined population Sample frame derived from population General public Congregation members Potential donors Potential adoptive parents 7 Identify survey targets

13. 8 Pick the right survey method to reach targets • Telephone • Mail • Online • Mall or store intercept

14. Telephone surveys • Advantages • Speed, in some cases • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Cost if you go outside • Hard to reach some socio-economic groups

15. Telephone surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. • $8,000 = $16 cost per interview • Review cost parameters • Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. • $4,000= $20 cost per interview • Review cost parameters

16. Telephone surveys • Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys. • $7,000 = $35 cost per interview

17. Doing your own telephone surveys • Sample list • Staff • Interns, volunteers, students, etc. • Call management system • Data recording system • Discipline • Persistence

18. Going outside for telephone surveys • Smaller local firms • Be flexible on timing

19. Mail surveys • Advantages • Lower cost • Easy if you have a mailing list • Easier to do yourself • Disadvantages • Slow • Lower control • Lower response rates • Need a mailing list

20. Doing your own mail surveys • Clear questionnaire layout • Sample list • Staff • Interns, volunteers, students, etc. • Lots of manual paper handling • Discipline • Patience

21. Online surveys • Advantages • Speed, in some cases • Control • Cost if you do it yourself (SurveyMonkey) • Disadvantages • Purchase a sample list • Unrepresentative of some socioeconomic groups

22. Online surveys • Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. • $3,300 = $6.60 cost per interview • Review cost parameters

23. Doing your own Online surveys • SurveyMonkey programmer • Sample list • Staff • Interns, volunteers, students, etc.

24. Mall intercept surveys • Advantages • Control • Probe and clarify • Cost if you do it yourself • Disadvantages • Slow • Need mall or store cooperation • Unrepresentative of some socioeconomic groups

25. Doing your own mall intercept surveys • Cooperation of mall or Wal-Mart • Staff • Interns, volunteers, students, etc. • Discipline • Patience .. Figure on one interview/hour

26. 8a Select an appropriate sample size • What are possible sources of error? • How far off could we be? • What sample size do we need?

27. 9 Give advance notice • If you know the target respondents • Newsletter • E-mail • Postcard • Benefit: improves response rate

28. 10 Remember you are asking for their time • This applies to… • Pre-survey letters or cards • Introduction to the survey • And the questions themselves • Don’t ask meaningless questions

29. 11 Write a good introduction to the questionnaire • Introduce yourself • “Research” not “survey” • Survey topic: social services • Not a sales call … not a fundraising call • You want their opinion / input

30. 12 Move from general to specific

31. 13 Lead with an interesting question • Engage the respondent • Ask interesting questions and people will talk

32. 14 Make judicious use of open-ended questions • Example: • “Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else?” • 2 to 3 per questionnaire maximum • Finish with a “final thoughts” question

33. 15 Use good response scales • Building a scale • Clearly distinguish good and bad • 1 Agree • 2 Disagree

34. Use good response scales • Building a scale • Add a polite positive and negative • 1 Strongly Agree • 2 Agree • 3 Disagree • 4 Strongly disagree

35. Use good response scales • Building a scale • Add a top positive response • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree

36. Use good response scales • Building a scale • Add a don’t know/no opinion • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Disagree • 5 Strongly disagree • 6 Don’t know / no opinion

37. Use good response scales • Building a scale • Add a mid-point if you really need it ... and balance the scale • 1 Very strongly agree • 2 Strongly Agree • 3 Agree • 4 Neither agree nor disagree • 5 Disagree • 6 Strongly disagree • 7 Very strongly disagree • 8 Don’t know / no opinion

38. 16 Make sure you cover all possible responses • Multiple choice questions

39. 17 “No opinion” does not equal “Don’t know”

40. 18 Always include a “refused” option

41. 19 Be careful with numeric responses • “About how much do you spend on gas each month” $__________ • Which of the following best describes the amount you spend on gas each month: • Under $25 • $26 to $50 • $51 to $100 • Over $100

42. 20 Be careful about “do you floss your teeth” items • Socially desirable responses • Responses that may imply guilt

43. 21 Test your questionnaire

44. Wrap-up • Begin with your objectives and information needs • Define the people you want to survey • Select an appropriate survey method • Write good questions • Test • Launch the survey • Analyze the results

45. Exercise