Using sound evaluation practices to support your project’s success

1. Evaluation for VISTAs Using sound evaluation practices to support your project’s success By Sara McGarraugh Improve Group Research Analyst

2. Let’s get to know each other

3. Overview • Purpose and benefits of evaluation • Evaluation tools that you can use • Designing an evaluation

4. What is evaluation? “In reality, (nonprofit) performance is all about translating caring, believing, and compassion into results.” • Letts, Ryan & Grossman High Performance Nonprofit Organizations 1999

5. What is evaluation? “ Assessing strengths and weaknesses …to improve effectiveness ”

6. Why Evaluation?

7. Demonstrate results *Fake data is used to create this dashboard

8. Reflect

9. Create a learning community

10. Influence others

11. Engage stakeholders

12. Plan for sustainability

13. What are the benefits of evaluation? • Knowledge and reflection • Information to share with stakeholders • Good evaluation practices are also good program management practices

14. Barriers to Evaluation • Humans resist change • We want to get along • Evaluation might suggest we are doing something wrong or would need to change • Time and cost

15. The Role of Evaluation Formative evaluation: How is the program delivered and is it meeting expectations?

16. The Role of Evaluation Summative evaluation: Did the program meet its goals?

17. The evaluation process

18. How are project and evaluation phases similar?

19. Putting Evaluation to practice

20. Start up: Define Purpose • Who are the key stakeholders of the evaluation? • What you hope to learn as part of the evaluation • Prioritize competing interests

21. Design & Planning: Refine goals S • What are your program’s aspirations (your goals)? • Goals should be SMART M A R T

22. Design & Planning: Define project outcomes What is the CHANGE you wish to see?

23. Practice: refining goals and outcomes

24. Logic models: a basic evaluation tool A Logic Model can describe how your program’s goals and activities lead to results and how to measure them

25. How are they useful? • Help get everyone on the same page • Encourage investment and buy-in • Facilitate organization • Provide clear and concrete guidelines • Serve as a roadmap throughout evaluation process • Useful for grant proposals

26. What does it show? OUTPUTS GOALS ACTIVITIES INTENDED OUTCOMES MEASURES

27. Definitions • Activities: • What your program does • Outputs: • Countable products showing an activity occurred (evidence) • Outcomes: • Benefit received from your program • Knowledge, behavior, condition changes

28. Example GOAL:For all students to be at or above their grade reading level OUTPUT: # of tutoring sessions # of books/articles read ACTIVITIES:-Tutoring -Theater -Book club INTENDED OUTCOMES:-Students are more confident in reading -Students think reading is fun MEASURES:-Survey -Check-in form

29. An alternative to a logic model

30. Practice: Logic model development

31. Break

32. Is it a tool, instrument, or protocol? Designing data collection tools

33. Design measures and tools • What type of evidence will demonstrate outcomes? • Design evaluation tools

34. Traditional data-gathering strategies

35. Consider creative ways to gather data

36. Matching evaluation tools to your work • Use available resources • Use knowledge of staff • Honor the wisdom of all of your stakeholders • Supported by infrastructure

37. Practice: Data collection tool design

38. Say what? If the question is a scale… “How valuable was your involvement with Sample Program?” • Very valuable • Somewhat valuable • Not valuable

39. Then, you can report outcome statements like: 65% of all respondents found their involvement with Sample Program to be very valuable – OR – Over half of respondents found their involvement with Sample Org to be very valuable. 80% of all respondents found their involvement with Sample Program to be somewhat or very valuable.

40. Say what? If the question is open-ended… “What was the most valuable part of volunteering for Sample Program?”

41. Then, you can report outcome statements like: Respondents frequently reported community engagement, connecting with participants, and learning about issues in the community as the most valuable part of volunteering with Sample Program.

42. Say what? If the question is a retrospective pre-test… “Please rate your opinion of the importance of volunteerism before AND after participating with Sample Program.”

43. Then, you can report outcome statements like: “85% of respondents rated volunteerism as very important after participating with Sample Organization compared to only 42% before participating.”

44. Types of questions What are the pros and cons of these question types?

45. Focus on simple and effective Photo by Heather McQuaid

46. Use parameters to make responding easier Photo by Oli Shaw

47. Avoid double-barreled questions, using jargon, or vague questions Photo by Nate Bolt

48. Practice: Survey or interview Question design

49. Implementation: Data gathering • Smile! • Practice in advance • Have a contact person for questions • Respect time & privacy • Create instructions

50. Survey administration • Timing • Response rates