Rebecca McQuaid rmcqconsult rebecca.mcquaid@gmail

1. Evaluation 101 Rebecca McQuaid www.rmcqconsult.com rebecca.mcquaid@gmail.com …a focus on Programs

2. About me… • Having an “evaluation brain” means… • Being curious! • Asking questions • Paying attention • Reflecting • Challenging our beliefs/ assumptions

3. About you… • What do you want to take away? • Exercise: Interests & Challenges • What do you bring to the table? • Experience with eval…

4. Eval 101 This presentation WILL address: Program Evaluation • Why, what and when … and some of the how • Backgrounder/ Refresher? This presentation will NOT address: Developmental Evaluation • Reporting (How to “tell the story”) • Capacity (Time, money, staff)

5. Why? EVALUATE … & INCREASE IMPACT! *Source: Evaluation for Organizational Learning: Basic Concepts and Practical Tools (Learning for Action Group)

6. Helps determine… • How a program is working • How successful a program is in achieving the desired outcomes • The impact a program has on the target group • Level of satisfaction* with the program • Program strengths and weaknesses • Cost effectiveness and efficiency of operation • At its heart, evaluation answers… • What is working? • 2) What is not? • 3) What difference is being made?

7. Program Evaluation • Goals & Objectives (“Why”) • Activities (“What”) • Anticipated Outcomes • Actual Outcomes • Conclusions (for learning!)

8. It can tell us… • Is a program working? • Achieving outcomes • Level of satisfaction • Strengths & weaknesses • Cost effectiveness • Efficiency • Impact on intended audience

9. Types of Eval • Formative • Begins during development and continues throughout program • How? • Needs Assessments • Process Evaluation

10. Needs Assessment • Takes place before program • Is there a need? • What is it? • First step in planning/ design • One-time analysis

11. Process Evaluation • Takes place during program • Assesses how program outcome(s) achieved • Collects data to improve program for efficiency & results

12. Helps determine… • Are processes used to achieve goals and objectives effective? • Was program carried out as planned? • What worked? Improvements? • Participant satisfaction with program

13. Types of Eval • Summative • Done at end of program • Focuses on success & effectiveness of program in reaching the stated objectives • How? • Outcomes-Based Evaluation

14. Outcomes-Based Eval • Were (pre-determined) outcomes met? • After implementation

15. Helps determine… • Impact on intended audience/ population? • Success in achieving outcomes? • Is there evidence to support continuation? Expansion?

16. PSA: Timing matters! “It’s much more effective to build evaluation in up front during the program planning process – this ensures that you gather the right data at the right time”

17. Types of Eval – Which one? • Formative • Needs Assessments • Process Evaluation • Summative • Outcomes-Based Evaluation Consider… type of program, purpose of eval, resource availability (people, $$), program timeframe

18. Ready to Evaluate? • Program Profile • Goals and Objectives • Inputs and resources • Program activities • Target audience/ populations • Tools • Program Profile • Logic Model CREATE A…

19. Logic Model • Visual of how a program (theoretically) works • Shows cause-and-effect relationship between activities and outcomes • Any problems or gaps in program? … evaluable?

20. Logic Model • For: Individual programs • Specific Detail • Activities, Outcomes • Implementation tool

21. Theory of Change • For: Organizations, initiatives • High-level overview • Strategies & Outcomes • Guiding/ Communication tool

22. Logic Model *Source: Evaluation for Organizational Learning: Basic Concepts and Practical Tools (Learning for Action Group)

23. Overarching goal: To decrease the rate of unemployment among young adults (aged 18-24) in the City of Kingston. E.g., Create posters for college & university campuses advertising career counselling program services E.g., Reduce avg. amount of time young adults spend in career counselling programs by 4 months E.g.,Two part-time staff with dedicated hours to young adult employment counselling E.g., Increase 1-on-1 career advising sessions with young adult clients to at least twice a week Decreased rate of unemployment among young adults in Kingston M

24. (Short term) Outcomes OBJECTIVES • Appreciate the role of evaluation in organizational learning • Understand different evaluation approaches, what they are for & when they are used • Use a Logic Model template to begin to organize a “picture” of your program

25. (Long term) Outcome GOAL • E.g., Develop capacity for participants in “Evaluation 101” workshops to plan theory-based program evaluations

26. Evaluation Framework • Outlines: • What will be monitored • How success will be measured • Who/where data will be collected from • How data will be collected

27. Data Collection Methods: • Depend on the type of data that needs to be collected and who data is being collected from Common ones: • Document review, Focus groups, Interviews, Surveys, Observation, Tracking/monitoring

28. Eval Framework (cont’d) • Analyze data & interpret results • Share results • Data collection tools are derived from the evaluation framework • Develop a data collection plan • Where will you get information? • When will it be collected? • From whom? How many people? • Etc. • Consistency = reliability

29. (Short term) Outcomes OBJECTIVES • Appreciate the role of evaluation in organizational learning • Understand different evaluation approaches, what they are for & when they are used • Use a Logic Model template to begin to organize a “picture” of your program

30. Evaluation 101 Rebecca McQuaid www.rmcqconsult.com rebecca.mcquaid@gmail.com …a focus on Programs