Bringing Evaluation In: Moving from Evaluation to Evidence-based Decision Support

1. Bringing Evaluation In: Moving from Evaluation to Evidence-based Decision Support Ben Pollard Director, VPS Portfolio Initiatives, UBC Former Director, Program Management and Evaluation, Immigrant Integration Branch

2. Immigrant Integration Overview • All facets of settlement and integration of newcomers • Settlement and orientation • Language • Labour market and Foreign Credential Recognition • Community integration • Community based, school-based, etc. • Diversity of immigrants (~40,000 annually) • 25+ program • 300 contracts with 100+ service providers • $140m annual programming budget

3. Context for Evaluation Significant Investment Multiple complex, diverse lines of business Multiple stakeholders High visibility Fed/Prov Agreement and Intergovernmental Relations Program review and re-procurement of all programming

4. Initial Evaluation model • Contracted out evaluation • Driven by external pressures • One at a time and intermittent • Minimal internal expertise to guide • Minimal linkage to decision-making processes

5. Challenges • One-offs • Lack of ability to compare • Lack of consistency • Lack of strategic or system evaluation • Simple evaluations • Didn’t address complexity • Funding bias • Consultant-driven methodology • Lack of internal guidance/overview/quality control • Lack of direct link between evaluation and decisions

6. New approach Internal evaluation team Reporting to same director as program managers External supports Transparency of methodology

7. Program Decision Support • Evaluations designed specifically to answer program and costing questions • Embedded evaluators - working closely with program managers • Triangulated recommendations – program managers and evaluators putting forward recommendations • Supports evidence-based decision-making

8. Strategic Evaluation Beyond program-specific How do these programs in total work to achieve our strategic goals? How do these programs work together? What are the broader pieces/horizontal approaches that impact on this suite of programs? How do those external partnerships work?

9. Building the Evidence base Testing service methodologies to be portable to other program areas/providers Comparative – this works better than that For which populations? Value for money Advanced analysis – e.g. what are the most important drivers of the desired outcome?

10. Putting Down Roots Model • Three key pathways have the strongest impact on recent immigrants’ sense of belonging and desire to stay in their communities. • Inclusive Community Path • Welcoming Community Path • Employment Equity Path (Recent Immigrants, n = 999) 54% of Variance Explained

11. Lessons Learned • Very effective at improving decision-making • Builds internal resources • Forces the internal conversation of evidence • Need to ensure transparency • Bring in outside evaluators where necessary • Skill sets • Data collection • Won’t work for all organizations

12. Wider implications What decisions do the clients need to be able to make? How can you support comparative funding decisions? What are the opportunities for building a wider evidence-base?

13. Questions? Ben Pollard ben.pollard@ubc.ca