Third Sector Evaluation: Challenges and Opportunities

1. Third Sector Evaluation: Challenges and Opportunities Presentation to the Public Legal Education in Canada National Conference on “Making an Impact” 26 October 2012 Michelynn Lafleche Director of Research, Public Policy & Evaluation, United Way Toronto

2. Evaluation in a changing context • Evaluation is a vital component of our work in the third sector – supporting organizational planning, decision-making, and learning • Evaluation has always played an important role in the sector, but the changing context of the third sector – the current funding environment and economic climate – has increased expectations regarding evaluation, with a particular focus on outcomes and accountability.

3. Funders and funded organizations need to work through 6 issues together Understanding the context in which our work is situated Acknowledging the limits of performance measurement and monitoring Recognizing the complexity of the interaction of individual, family, community and structural factors that influence outcomes Addressing the question of attribution Building evaluation capacity across the sector Planning for evaluation as part of program implementation

4. Challenges of evaluation focused on outcomes and accountability (1) • Results in data-driven exercises that do not always contribute to a meaningful assessment of impact or to program improvement • Generates vast quantities of data, some of questionable value • Shifts the focus among service providers from service delivery to measurement with no clear benefit to the programs

5. Challenges of evaluation focused on outcomes and accountability (2) • Common outcome setting processes, often driven by funders, are becoming more common-place, but outcome reporting is most useful when outcomes are aligned to the program theory of an intervention and the mission of an organization • Outcome measurement alone can only demonstrate if something is changing, not why it is changing or if the intervention or initiatives is causing the change • Linking outcomes to funding decision may intensify these challenges

6. Opportunities to advance evaluation • Moving beyond outcome evaluation to include not just a focus on what worked, but why it worked, with whom and in what context – i.e. community-based approaches such as developmental and realist evaluation or participatory approaches using qualitative methods • Promoting evaluation for learning to explore how to understand and solve problems more effectively • Fostering opportunities for knowledge generation and exchange about what does and what doesn’t work • Recognizing that evaluation is not a solution all on its own

7. Characteristics associated with evaluation leaders • A long-term commitment to evaluation • An appreciation that evaluation is an ongoing and evolving activity • A willingness to share mistakes and learn from failure • An engaged and committed team of internal staff, outside evaluation expertise, and key stakeholders • Clear leadership commitment to sharing and using evaluation findings to support learning and improvement • A commitment and process to translate findings into action, knowledge exchange, and advocacy • An ability to tailor evaluation to context • A focus on building the capacity of all partners

8. Contact details Michelynn Lafleche Director, Research, Public Policy & Evaluation United Way Toronto mlafleche@uwgt.org 416 777 1444 x220