Reflections on Realist Evaluation: Unpacking the Mechanisms of Change

1. Reflections on Realist Evaluation:Unpacking the Mechanisms of Change Marnie Carter Ned Hardie-Boys Ella Spittle

2. What we will cover • Overview of realist evaluation • Principles and approach • Implementation of a realist evaluation • Lessons from a realist evaluation • Questions and discussion

3. What is realist evaluation? Realist evaluation is an approachthat can be broadly applied to different types of evaluations of policies and programmes The term was drawn from Pawson & Tilley’s book, Realistic Evaluation (1997)

4. Why realist evaluation? • It addresses the limits of traditional approaches, which • Only ask “does this programme work?” • Assume that programmes can be credited with general cause-and-effect statements • Realist evaluation focuses deeper on why a programme works • “What works for whom, in what contexts, and how?”

5. How does realist evaluation work? Realist evaluation focuses on • Mechanisms: Why a programme ‘works’ (or does not work) • Context: What conditions are required for the mechanisms to work • Outcomes: The results of different mechanisms working in different contexts

6. Mechanism + Context = Outcomes

7. Principles of realist evaluation I • Programmes, policies and interventions address problems by influencing change • Change is instigated by allowing participants to make different choices • Making different choices requires changing resources and/or reasoning • The combination of resources and reasoning allow participants to change their behaviour (the mechanisms)

8. Principles of realist evaluation II • Changes in behaviour will vary depending on participants’ context • Context, therefore, affects which mechanisms are triggered and what outcomes are achieved • Outcomes are varied because different mechanisms are triggered in different contexts • Variation is why realist evaluation findings are always provisional: “What works for whom, in what contexts, and how?”

9. Realist evaluation in practice Evaluation of changes to the Road User Charges (RUC) system • Examine initial impacts • What mechanisms have been triggered? • Evaluate outcomes • Has activation of those mechanisms resulted in the expected outcomes? • Recommendations • How can the system better influence activation of which mechanisms, by which stakeholders, in which contexts, to achieve desired outcomes?

10. RUC system – background • RUC is a tax for road “wear and tear” • A number of changes were made, but the main change was to reduce complexity and opportunities for evasion • The previous RUC system charged road users based on a vehicle’s net weight • The new RUC system charges road users based on the maximum potential net weight

11. Evaluation design • Two stage • Framework • Implementation • Why a realist approach? • Steps • Theory-building (outcomes and mechanisms) • Questions, measures, methods and sources

12. Efficiency I Mechanisms Behavioural response to simplify admin processes Organisational response by NZTA to simplify admin processes and streamline payment methods Organisational response by NZTA to reduce complexity of investigations to recover unpaid RUC Outcome 1. Reduced compliance costs for operators 2. Reduced administrative complexity for government

13. Efficiency II Mechanisms Incentive for operators to load vehicles up to their RUC weight Organisational response by NZ Police to reduce officer time spent on enforcement Simplified processes and reduced costs incentivise new providers to enter market Outcome 3. Improved efficiency in vehicle use 4. Simplified enforcement of RUC 5. Increased uptake of electronic systems

14. Integrity I Mechanisms Deterrence: greater chance of being caught changes risk perception Deterrence: perception of consequence of being caught leads to change in behaviour Deterrence: requirement to produce EDR acts as a deterrent Outcome 6. Reduced RUC evasion

15. Integrity II Mechanisms Clarification/ communication of changes to system Monitoring and evaluation of the system Outcome 7. Increased understanding of the system

16. Equity Mechanisms Organisational response by government to monitor impacts and ensure no disproportionate effects on operators Outcome 8. Revenue neutrality within groups of vehicles (operators pay fair share)

17. Cost recovery Mechanisms Penalties are sufficient to incentivise people to pay on time Detection: NZTA receives info from WoF providers and is able to identify evasion Investigative: the issuing of binding assessments assists in recovery Outcome 9. Reduced late payment 10. Improved recovery of unpaid RUC

18. Implementing the RUC evaluation • Evaluation questions placed emphasis on behavioural change • What impact have the RUC changes had on the level of evasion? • To what extent have the RUC changes contributed to changes in operator perception and behaviour regarding evasion? • Survey and interview tools were designed to seek evidence of behavioural change • Analysis focused on • whether the expected mechanism(s) had been activated • for whom, how and in what context the outcome had been achieved

19. Example: efficiency in vehicle use • Previous system: transport operators purchase RUC for the exact weight they will carry • Policy change: Fixed ‘RUC weight’ for vehicles • Mechanism: Incentive for operators to load vehicles up to their RUC weight • Expected outcome: Transport operators will use their vehicles more efficiently • Required context: transport operators are able to, and do, change their business practices

20. Evaluation findings • Commercial transport operators were already loading as efficiently as possible • Other considerations (safety, customer needs) were more important than RUC in loading decisions • Many loads ‘cubed out’ before reaching the maximum weight • Vehicles tend to be load-specific making it difficult to ‘back load’ The mechanism was not activated

21. Example: Recovery of unpaid RUC • Previous system: time-intensive manual process for NZTA to identify unpaid RUC amongst light diesel vehicles; opportunity cost • Policy change: Odometer reading reported as part of Warrant of Fitness inspection • Mechanism: NZTA is better able to detect evasion • Expected outcome: Unpaid RUC will be identified and recovered • Required context: NZTA develops a system to receive data from WoF providers, and uses the information to recover debt

22. Evaluation findings • NZTA developed an automatic reconciliation and invoicing system • The amount of invoiced debt has increased substantially • Only 14% of invoiced debt has been recovered The mechanism was partially activated

23. Reflections

24. It can be easy to get bogged down labelling mechanisms

25. Lesson one Focus on describing mechanisms and what is needed to activate them – not naming them

26. The perils of tunnel vision

27. Avoiding tunnel vision • Expected outcome of reduced compliance costs for transport operators • Transport operators were expected to modify their administrative processes in response to a simpler system • An online survey and case studies were used to seek evidence of behavioural change • No evidence that operators had changed their behaviour was found Initial conclusion: outcome was not achieved

28. However… • Changes were made to the NZTA RUC administration fee structure • 30% drop in NZTA admin fees (although this was a small portion of overall RUC compliance costs) • Transport operators had therefore achieved a small reduction in cost, despite not changing their behaviours Revised conclusion: the outcome was partially achieved

29. Lesson two Behavioural change of the intended recipients is not the only means through which outcomes can be activated

31. Reduced evasion of RUC • Previous system: opportunity for operators to under-nominate weight; outdated penalties for other forms of evasion • Policy change: Fixed RUC weight; updated offences and penalties regime • Mechanism: The removal of weight-based evasion is built into the system (no mechanism); updated penalties act as a deterrent to other forms of evasion • Expected outcome: Evasion will reduce • Required context: Transport operators are aware of the new penalties, which causes their risk perception to change resulting in modified behaviour

32. Evaluation findings • Police Heavy Vehicle Operation data showed that evasion had reduced from 4% in 2012 to 1.2% in 2013 • No transport operators we interviewed admitted that they had previously evaded RUC and now were not • We were therefore unable to identify how and why this effect had occurred (other than through speculation)

33. Lesson three For sensitive or morally questionable topics, a focus only on what (rather than how and why) is appropriate.

34. Overall reflections • The realist evaluation approach led us to lines of enquiry we may not have otherwise considered • Clearly defining the mechanism of change assisted us to frame what evidence we were looking for • Provided a means to demonstrate how and why the changes to the RUC system ‘worked’ • Helped to get inside the black box • A useful tool for the evaluation kete

35. mcarter@allenandclarke.co.nz nhardie-boys@allenandclarke.co.nz espittle@allenandclarke.co.nz