Evaluation design and implementation

1. Evaluation design and implementation Puja Myles Puja.myles@nottingham.ac.uk

2. Session outline -Evaluation frameworks -CDC framework for evaluation -Theory of change and logic models -RE-AIM framework -Maxwell’s quality assessment framework -Practical exercise: Using a logframe matrix and decision models for evaluation planning/design

3. F RAMEWORK Step 1 Step 2 Deciding and measuring health outcomes Step 3 Step 4 What is an evaluation framework?

4. CDC framework for evaluation Step 1: Engage stakeholders Step 2: Describe the program Step 3: Focus the evaluation design Step 4: Gather credible evidence Step 5: Justify conclusions Step 6: Ensure use and share lessons learned

5. Step 1: Engage stakeholders Key stakeholders: • People involved in programme operations (funders, managers, administrators) • People served or affected by the programme (clients, family members, elected officials, sceptics) • Primary users of the evaluation (will be a subset of all the stakeholders identified; these are the people who can act on findings and bring change)

6. Role of stakeholders • Clarify the programme objectives • Help you elucidate the underpinning theory of change • Help design and carry out the evaluation • Help frame recommendations for practice based on findings • Initiate change/act on recommendations i.e. ensure that the evaluation is meaningful

7. Step 2: Describing the programme-1 -Mission and objectives of the programme-The problems addressed by the programme (nature and magnitude of the problem; populations affected)-How the programme intends to address the problem (theory of change)-Expected effects of the programme

8. Step 2: Describing the programme-2 • Activities • Resources • Context (setting and environmental influences e.g. Political/historical/social) • Logic Model

9. Theory of change • This approach involves setting out the series of outcomes that are expected to unfold as a result of the various components of the intervention as a basis for planning the evaluation strategy. • Can be visualised as a sequential process of ‘if-then’

10. Logic model/logframe matrix • A practical approach to understanding the theory of change for a given intervention • Can be used with stakeholders

11. An example logframe matrix

12. Step 3: Focusing the evaluation design Things to consider: • Purpose of evaluation (feasibility, effectiveness, change, empowerment, sponsor requirement) • Evaluation questions (merit, cost-effectiveness, equity, quality) • Feasibility • Ethics

13. Study designs Ovretveit (1998) outlined six basic evaluation designs: • Descriptive • Audit • Outcome (the before-after comparison; quasi-experimental design) • Comparative experimental • Randomised controlled experimental • Intervention to a service (impact on providers and patients)

14. CDC framework: Steps4-6 Step 4:Gather credible evidence (what outcomes and how will you measure these) Step 6: Justify conclusions (attribution versus contribution; alternative explanations such as bias, chance, confounding) Step 7:Ensure use and share lessons learned (stakeholder involvement; participatory approaches)

15. RE-AIM framework for measuring public health impact Glasgow et al (1999): • Reach (uptake; who benefits; who is left out) • Efficacy (include behaviour outcomes and participant-centred quality of life measures; consider both positive and negative outcomes) • Adoption (proportion & representativeness of settings): use direct observation, interviews, surveys • Implementation (the extent to which a programme is delivered as intended); audit • Maintenance: long-term maintenance of behaviour change (both clients and service providers)

16. Assessing Quality Maxwell’s dimensions of health care quality: • Access to services • Relevance to need (for the whole community) • Effectiveness (for individual patients) • Equity (fairness) • Social acceptability • Efficiency and economy