The intrepid researcher: Evaluating programmes and social interventions in public services

1. The intrepid researcher:Evaluating programmes and social interventions in public services Mary Dixon-Woods Department of Health Sciences University of Leicester

2. Why intervene • Because there is a problem to be addressed • Because there is a reasonable basis for thinking that it can be addressed, for reasonable cost and with minimal damage • There are good reasons for believing the intervention is better than current situation • There is some theoretical or empirical basis for those beliefs

3. Why evaluate • To find out what the interventional programme actually was (not what it was intended to be) • To identify the effects of the programme (good and bad) • To produce evidence of costs and value for money

4. Why evaluate • To figure out how and why the programme produced the effects it did • To determine whether to continue with a programme/replicate it elsewhere • To make modifications and adaptations to improve the programme • To satisfy needs for accountability

5. Foot care in England • There are over 70 amputations of feet a week in England, of which 80% are potentially preventable. • In 2007/8, nearly a quarter of people with diabetes did not have a foot check. • You are twice as likely to have your foot amputated if you live in the Southwest compared with the Southeast.

7. So – improvement is needed! But how do we intervene? Should we: • Send an email around to remind GPs to do foot checks? • Train GPs in doing foot checks? • Pay them to do the checks? • Fine them if they don’t do them? • Give them a goal they have to meet for their annual performance review? • Give them feedback every month on how they compare with others? • Publish a league table of amputations by GP? • Tell patients they should ask their GP for a foot check?

8. What tends to happen in social interventions in public services • Bright idea /recommendation from policy push/ “latest thing” • Borrowing or building on a currently fashionable or politically favoured model – e.g. “tough accountability” • Theory of change not explicit

9. What tends to happen in social interventions in public services • Not clear what process is being targeted • No attempt to expose to systematic challenge or understand how it works • No search for unanticipated consequences or toxic effects • Poorly described or at wrong level of specification, so impossible to reproduce

10. Provision of fruit and veg a school Lower obesity in children

11. Provision of fruit and veg a school Lower obesity in children

12. Programme theory “Nothing improves research design so much as having a clear idea about what is being investigated. An important function of theory in research design is to help researchers ensure that they are playing in the right ballpark to begin with— that is, to help them avoid studying the wrong thing” (Lipsey, 1993)

13. What is programme theory Plausible model of how the programme works • Makes explicit the assumptions different parties hold • Identifies the desired outcomes • Identifies the activities and resources, and who will undertake them • Identifies the mechanisms that link those activities and resources to the desired outcomes • Identifies conditions likely to be favourable or unfavourable to functioning of the mechanisms • Identifies possible unwanted outcomes (the side effects)

14. Programme theory • Can be expressed as a narrative • Or as logic models/diagrams Inputs Outputs Outcomes - impact Sitation WWhat we invest What we do What the results are

15. Evaluative causal analysis: three components • A logical component that allows justification that the causal link is reasonable/plausible • Methodological (design) component capable of enabling reasonable inferences about the extent to which outcomes can be attributed to the program actions • Empirical component that produces evidence of process and outcomes

16. Programme theory • “All three of the components of causal analysis are importantly and substantially strengthened by an explicit theory about the nature and details of the change mechanism through which the cause of interest is expected to produce the effect(s) of interest”. • But 70% of studies offered either no theory or only general statements of programme strategy/principles. • Lipsey, 1993

17. Without a decent programme theory • Programme logic may be flawed • Programme may not be causally linked to desired outcomes • The implementers may be blamed if the problem is not solved • It’s hard to replicate the programme or design new, similar ones

20. Explaining Michigan’s success

21. Theory • Program theories are “small” theories • Not big theories of social phenomena • Not doctrines • May be multiple • Subject to continual re-specification and improvement

22. Specifying the theory • What is the problem you are trying to address? • What is the rationale that supports the way you are trying to tackle it? • Why should it work? Why should it work better than what’s already happening? • What are the distinctive features of the intervention? What makes them distinctive?

23. Reviewing the theory • Are program goals well defined? How will we know if they have been achieved? • Are the program goals feasible and realistic? • Is the change process plausible, with no significant conceptual and operational weaknesses? • Is there a good operational plan? Are the components, activities and functions well specified?

24. How to build the programme theory • Off the shelf theory + previous studies provide a point of departure • Programme personnel are working with a theory (even if they don’t know it) – which you need to elicit e.g. through interviews • Programme documents • May produce a causal diagram/logic model or a narrative that links the components

25. Who cares about the programme? • The funders • The people providing the programme • The people implementing the programme locally • The intended beneficiaries • Others serving the same beneficiaries

26. Kinds of questions we might ask • If the programme is going really well, what will we see? Why? • If the programme is going just ok, what will we see? Why? • If the programme is floundering, what will we see? Why?

27. Programme designers’ theory as elicited

28. Elicited theory

29. Elicited theory

30. Elicited theory

31. Challenges of building programme theory • Elicited theory is often incomplete or not fully coherent • It sometimes reflects an attempt to please or appease various parties • It is often weak on key links • Operational links in the chain • Logical and conceptual linkages

32. Challenges of building programme theory • Need to think about values and interests • Whose definition of success gets to count? • What are the responsibilities of evaluators?

33. Programme in action may look very different from programme as intended • Key activities may not happen or look different from how expected • The mechanisms may not function in the way intended • External and internal contexts may powerfully modify what is possible • Programme designers may improvise, innovate and adapt • So part of the job of the evaluator is working out what really happens, and updating the theory

34. Evaluating the programme in action • May need multiple methods (quant and qual) to figure out • What the programme really is • How it works (mechanisms) • What effects (both good and bad) it has • Whether it operates differently in different contexts

35. Evaluation designs • Need to choose design carefully, using programme theory and nature of answer wanted as a guide • Before and after studies • Controlled before and after studies • Prospectively designed experiments • Mixed methods • Qualitative • Case studies

36. What is deemed “evaluable”? • Perceptions of relevance • Just the goals of the program – or side effects too? • How it works as well as whether it works? • Perceptions of credibility • Perceptions of threat • Who really wants to know the answer? • Difficulties of accessing data on some measures • Costs or burdens of gathering data • Ethical issues

37. Using team ethnography • Team of observers + non-observers • Allows multiple observations • “Blitzing” or multiple observations over time, documentary analysis, interviews, chats • Allows for integration of multiple perspectives, esp through team debriefing • Ideally integrated with quantitative measurement

38. A programme to improve patient safety • New system for identifying patients at risk of blood clot • Process measures - are forms filled in? • Immediate outcomes – do higher risk patients get preventive treatment? • Ultimate outcomes – are fewer patients harmed? • Mediators and contexts – type of ward, type of patient, training of staff etc • Side effects – distracts staff, patients ignored once assessed as low risk, diversion of resource

39. A programme to improve patient safety • New system for identifying patients at risk of blood clot • Process measures - are forms filled in? • Immediate outcomes – do higher risk patients get preventive treatment? • Ultimate outcomes – are fewer patients harmed? • Mediators and contexts – type of ward, type of patient • Side effects – distracts staff, once assessed as low risk ignored, diversion of resource

40. A training programme to reduce racism • New system for training teachers • Process measures - • Immediate outcomes – • Ultimate outcomes – • Mediators and contexts – • Side effects –

41. Measuring • Measuring often produces changes in the activities you are monitoring, in addition to any programme effects • Risks of • Not knowing whether your programme worked • Goal displacement • Gaming

42. Data for performance management • Centralised performance management of public services • “Targets and terror” regime throughout 2000s • Organisations and professionals have become adept at working out what they need to do to survive

43. Measuring – unwanted consequences • Incentives to respond to the measures rather than the intentions behind them • Tunnel vision • Suboptimisation • Myopia • Measure fixation • Gaming • Failure to capture subtle changes

44. Goodhart’s law “Any observed statistical regularity will tend to collapse once pressure is placed on it for control purposes”

45. Interpreting an evaluation • Important to distinguish how the interventions worked from how the programme worked • It is possible for the interventions to be effective but not the programme

46. Challenges with progamme-theory guided evaluations • Evaluators get in the way of programme • May try to recover sunk cost of the programme theory they developed, and be unwilling to consider other possibilities • Theory may be completely wrong – programme may work but for reasons other than those hypothesised by evaluators

47. Completing an evaluation • Very often the results of programmes are more modest than hoped for; sometimes a null result or negative • Sometimes the findings are uncomfortable or threatening for various stakeholders • Evaluators are often in multiple principal-agent relationships • Need to plan well in advance for communication and dissemination

48. Who cares? • “Administrators’ interests in organizational stability, budget maximisation, and the promotion of a favorable image, contribute to a general desire to prefer evaluations and performance reports that do not cast programs in a bad light.” • Schwartz and Mayne, 2005

49. Evaluating public services • Social science evaluation in public sector organizations has an important role • But such studies may be challenging or risky for the staff being studied

50. Architectures of ethics control • Institutionalised as ethical review by committee, modelled on medical sciences • Organised to produce an audit trail • Distinctive for the emphasis on • immutable principles as a basis of decision-making • construction of the research participant as vulnerable and potential victim • Valorisation of informed consent