Erica Wimbush Head of Evaluation, NHS Health Scotland

1. Getting the most out of evaluationReflections on achievements and challenges in commissioning and managing evaluations -A Scottish perspective Erica Wimbush Head of Evaluation, NHS Health Scotland Swiss Jubilee Symposium on Public Health Evaluations – Lessons Learned and Future Directions 17 June 2011, University of Berne

2. Scotland in Europe What is distinctive? What do we offer? • A small country in Europe • Strong tradition of public health • Focus: Health inequalities, deprivation, rurality • Public health located within NHS but strong emphasis on ‘partnership working’ • Data collection of a global standard

3. 60 staff, €6.7 million 300 staff, €30.3 million The national public health agency UK Devolution from 1997 – divergent roles Health Education Board for Scotland (1991-2002) (2003- present) Role: Provides specialist advice to Government on improving Scotland’s health and reducing inequalities and supports policy and programme implementation at local level

4. Health Scotland: Evaluation Team • Our mandate: ‘To coordinate the evaluation of health improvement policy in Scotland’ Health Improvement: The Challenge, Scottish Govt 2003 • Our role: Enables a better understanding of the reach and impacts of policies and actions through outcome-focused planning, monitoring and evaluation. 3 main areas of work: • Commissioning and managing evaluations • Supporting others to do outcome planning, evaluation and performance reporting • Evaluation Summer School & workshops Key principle: Utilisation-focused evaluations

5. Public health evaluation 75% of evaluations of health improvement interventions are commissioned by Health Scotland, the Scottish Government or local NHS boards Health Scotland evaluations: • Policy evaluations • Programme evaluations • Local project evaluations

6. Achievements & Challenges

7. Improving Evaluation Quality What are the Quality issues? • Research design and methods • Timeliness and relevance for decision-making • Responsiveness, flexibility, creativity What worked? • Development of theory-based evaluation practice • Establishing a vibrant evaluation community • Explicit quality standards and competencies (work in progress) Continuing challenges • ‘Institutionalisation’ of evaluation – part of the culture but no independent budget or requirement

8. Independence vs Utilisation What is the issue? • Maintaining independent position viz a viz policy/program can be in tension with commitment to stakeholder involvement What worked? • Intermediary ‘bridging’ role Continuing challenges • Recognising the un-evaluable!

9. Evaluation Capacity & Culture What is the Capacity-building issue? • Lack of confidence and competence (practitioners) • Intelligent demand (funders) • No recognized training or qualification (evaluators) What did we try? • Workshops, training, toolkits • Importing talent from abroad • Most effective – policy move to outcomes management & accountability across public services Challenges • Extension of team support role and skills • Sustaining the gains if/when political environment changes

10. Commissioning &/or Coordinating? What is the issue? • Reduced funding available • Commissioning focus – single study perspective • Quality issues – making more of in-house expertise What worked? • Evaluation strategy – shift evaluation focus to policy • Coordination of a portfolio of inter-linked studies Continuing challenges • In-house capacity - Further extension of role • Internal infrastructure to support data collection and analysis

11. Future – Sharing practice What do we need? • Qualifications • Standards & Competencies What can we bring? Theory-based evaluation practice Vibrant evaluation community Bridging role