Introduction to Evaluation Techniques

1. Introduction to Evaluation Techniques Puja Myles Puja.myles@nottingham.ac.uk

2. Module outline Session 1: Introduction to evaluation techniques Session 2: Evaluation design and implementation Session 3: Participatory approaches to evaluation Session 4: Economic evaluation: the basics Session 5: Economic Evaluation: types and costs Session 6: Economic Evaluation: Outcomes and uses Session 7: Prioritisation exercise

3. Module Learning Outcomes At the end of this module you should be able to: • Recognise the basic principles of healthcare evaluation • Identify the key features of different types of economic evaluation • Distinguish between various evaluation types and select the appropriate type for a given scenario • Choose an appropriate outcome measure for a given evaluation question • Propose a suitable study design for a given evaluation problem • Design an evaluation drawing on the theory of change and logic model • Apply evaluation principles to prioritisation of healthcare

4. Module Assessment Write a protocol for an evaluation of either: • A new or existing service you work with OR 2. One of the examples provided Your evaluation can look at the whole service or focus on particular aspects. It MUST include some economic evaluation, as well as looking at the effectiveness and quality of the service

5. What is Evaluation? At its most basic, evaluation is concerned with assessing whether interventions are effective…

6. Defining Evaluation-1 “The purpose of evaluation is to measure the effects of a program against the goals it set out to accomplish as a means of contributing to the subsequent decision-making about the program and improving future program-making” -Weiss (1972)

7. Defining Evaluation-2 “The aim of evaluation is to contribute towards solving practical problems, in terms of what works and why. It is about collecting information to inform action. Most of all it is about learning from experience.” -Springett (2001)

8. Types of evaluation- 1 Ovretveit (1998) categorised evaluation depending on when the evaluation is carried out in the healthcare intervention planning cycle: • Formative/developmental evaluation • Process evaluation • Summative evaluation

9. Types of evaluation- 2 Green and South (2006) outline different models of evaluation based on the purpose of the evaluation: • Goal-oriented evaluation: effectiveness, efficiency and economy of an intervention • Decision-orientated evaluation: improve decision-making • Evaluation research: providing explanations for outcomes • Responsive evaluation: process of evaluation and perspectives of evaluation • Goal-free evaluation: Openness to achievements other than those prescribed by the intervention’s aims and objectives

10. Evaluation, Research or Audit? ‘The use of evaluation within a research context is distinct from its use within other management activities such as quality assurance and monitoring. These are applicable only to a particular context, whereas research aims to be generalisable across different contexts’ -Smith et al (2005)

11. Audit • Useful in quality assurance of a process • Compare practice against a set of standards • When you know that the process will predict outcomes provided it is in keeping with the accepted standards • Example: Audit of prescribing (you know the drug works provided prescribing guidelines are followed)

12. And what about Monitoring? ‘Emphasis on recording what has happened in terms of programme delivery, whereas evaluation is concerned with assessing what has been achieved and how any changes have come about’ -Green and South (2006)

13. How are evaluation questions commonly framed?

14. Efficacy, effectiveness and efficiency* • Efficacy- does an intervention work in ideal conditions? • Effectiveness-does an intervention work in real life? • Efficiency-ratio of useful output to total input; incorporates the notion of minimising waste. Often phrased as ‘value for money’ or ‘return on investment’

15. External vs. Internal evaluators? What do you think?

16. How can you make an evaluation meaningful for practice?

17. Stakeholder involvement • Identify key stakeholders • Recognise potential conflict of interests (multi-agency funded projects) • Specify aim of evaluation (e.g. to assess overall improvement in a given health outcome; or quality of service; or value for money…)

18. Measuring health outcomes

19. Influence of Health Perspectives 1 Biomedical Model of Health • Health = Absence of disease (pathology) • ‘Functionalist’ approach • Objective, numerical measurements based on ability to function • Preconceived assumptions about individual perceptions of health

20. Influence of Health Perspectives 2 Social Model of Health • ‘Disease’ and ‘Illness’ different concepts • WHO definition of Health • ‘Hermeneutic’ approach to measure aspects of health unique to individuals • Open-ended questions and qualitative

21. Qualitative versus Quantitative Quantitative - quantifiable • reliable • generalisable (? possibly) • Decontextualises processes Qualitative -provides context -understanding of the ‘whys’ -generalisability is a major issue

22. Health Outcomes- examples • Health Behaviours • Adherence to treatment • Specific Diseases • Depression • Emotional balance or well-being • Global health status using EQ-5D, SIP or SF-36 • Self-reported health status

23. Health Outcomes? • Self-confidence • Self-efficacy • Social involvement • Sense of achievement/direction • Stress relief • Participation in social networks

24. Introducing EQ-5D • Standardised measure of health status • Generic measure of health • Can be used to generate quality of life measures • 2-page self-completion questionnaire • Need to register study with EuroQol: http://www.euroqol.org/home.html • May have licensing fees depending on study Do you think this is a good measure?

25. Introducing the SF-36 • Shorter variants like the SF-20 and SF-12 are available • Well-validated • Can be used to generate quality of life measures • Is available free in the public domain from RAND http://www.rand.org/health/surveys_tools.html