Accountability in Health Promotion: Sharing Lessons Learned

1. Accountability in Health Promotion: Sharing Lessons Learned Presentation to the Canadian Evaluation Society June 2, 2003 Vancouver, British Columbia Management and Program Services Directorate Population and Public Health Branch Health Canada

2. Presentation Objectives • Share lessons on the process and planning elements of an evaluation synthesis for health promotion programs: • Evaluation environment and review ‘Umbrella’ approach • Methodology/framework • Transferable knowledge developed on tools and approaches 2 Health Canada, Population and Public Health Branch

3. Federal Context: Measuring for Results • Increased role for accountability through ‘Results based’ management • Shared accountability/horizontal approach • ‘Value for money’, relevance, management practices • Increased role for prevention/promotion in health policy • Evaluators faced with dual task of developing performance measurement strategies while substantiating health promotion outcomes 3 Health Canada, Population and Public Health Branch

4. About the Initiative • Obligation to renew authorities under which Grant and Contribution health promotion programs are administered • Improving program management practices and knowledge base • Evaluation Synthesis for six programs • ‘Umbrella’ Results-based Management and Accountability Framework for 15+ programs • Horizontal initiative engaging many stakeholders 5 Health Canada, Population and Public Health Branch

5. What is the Synthesis? • Summary of evaluations – their methodology and results • Integrates findings based on topic/functions, versus looking at practices and outcomes through a single program lens • ‘One window’ approach to view net effects of programs (evaluation practices and program outcomes) 6 Health Canada, Population and Public Health Branch

6. Synthesis vs. Single Evaluation • Bias unchallenged • Time-specific, focus on outputs vs. rigour and data quality • Activity/project/issue oriented • Transferable frameworks • Less opportunity for bias in results; add strength to overall results • Future application for improving evaluations • Systematic, comprehensive, allows for comparability of large amounts of information • Combines formative and summative • Enhance credibility of process and product, clearer link to policy Health Canada, Population and Public Health Branch

7. Basic Steps: Evaluation Strategy • Build the Framework • Build the Team • Develop the Tools 7 Health Canada, Population and Public Health Branch

8. Build the Framework • Identify stakeholders and built synthesis review team • Develop ‘umbrella’ framework and focus the evaluation • Propose a research plan (define and pose questions) using common assessment criteria • Establish tools for gathering and analyzing evidence • Establish consultative process 8 Health Canada, Population and Public Health Branch

9. Build the Team • Build team and assign functions • Management team oversees process • Functional task group plans and implements • Multi-disciplinary research team • Assign primary and secondary reviewers • Maintain regular communication and document experience • Ongoing validation of review process and outcomes being reported Health Canada, Population and Public Health Branch

10. Build the Team and Assign Functions Executive Management Project Authority Project Management Team Middle Management Task Groups Renewal Evaluation Risk-based Audit Program/ Functional Managers • Planning & Product development • Communication, messaging, capacity building • Systematic implementation 9 Health Canada, Population and Public Health Branch

11. Develop the Tools • Evaluation Synthesis framework and work plan • Evaluation synthesis review protocol: • Literature review of key supporting documents • Assessment of review instrument through pilot • Instrument validation • These tools and the Evaluation Synthesis will be used for the ‘RMAF’ 10 Health Canada, Population and Public Health Branch

12. Preliminary Findings: Lessons • Process • Planning intensive • Appetite for reorienting evaluation culture • Consensus driven approach building trust into process • Creating momentum to build evaluation further into program management practices • Reports • Output (vs. health outcome) oriented due to context • Method/data source quality and availability • Guidance on evaluation Health Canada, Population and Public Health Branch

13. Added Value • Developing common assessment tools with broader applications: • Evaluation Synthesis Review ‘Protocol’ • Evaluation Framework/Methodology • Standard indicators to support integrated approaches • Consultation workbook • Synthesis research on methods & interventions • Documenting experience as a transferable approach • Building a ‘community of practice’ and culture of evaluation to be more collaborative • Strengthening the evidence base linking to policy and supporting integrated health promotion strategies 13 Health Canada, Population and Public Health Branch

14. Project Challenges • Characteristics of population health promotion • Level of implementation and ability to prove ‘effective’ • Summarizing key achievements of programs (e.g. time, resources, variety) 11 Health Canada, Population and Public Health Branch

15. Opportunities • Establishing a comprehensive, collective, integrated and structured evaluation strategy • Taking stock of evaluation methods and techniques in health promotion; their limitations and possibilities • Developing integrated tools and mechanisms for accountability to improve program design and delivery 12 Health Canada, Population and Public Health Branch

16. The Way Forward • Share lessons learned on process and products: • Results of synthesis report • Results of ‘RMAF’ • Develop generic synthesis model to validate assessments of management practices and their evaluations • Maintain focus on building capacity in evaluation… 15 Health Canada, Population and Public Health Branch