Intervention Logic

1. Intervention Logic A Presentation to the Pathfinder Project Karen Baehler Victoria University of Wellington 463 5711 karen.baehler@vuw.ac.nz

2. The problem Citizens want to know if government is making a difference. Are we getting results in return for our taxes? Ministers want better advice: “The most common problems which can be discerned in recent experience are: • overstatement of what will be achieved; • under-explanation of how policy actions will achieve the claimed outcomes.” From Improving Policy Advice (1993) by G. R. Hawke, p. 27 Victoria University of Wellington

3. The solution • Identify goals (outcomes) • Chart a course to those goals • Measure current progress • Stop things that don’t work • Alter things that sort of work • Keep improving things that do work • Discover/invent new avenues to success Victoria University of Wellington

4. Outline • What is intervention logic? • What are its prerequisites? • What are its uses? • What are its blind spots? • How do we minimise the blind spots? • How do we know if an IL is working? Victoria University of Wellington

5. What is intervention logic? • A testable theory of causation • Linked “if-then” statements • Action/reaction pairs • A chain of conditions to be achieved • Ultimate/end outcome = policy goal • Intermediate outcomes and immediate impacts • Lead to the end outcome • But are not ends themselves • A basis for confirming performance Victoria University of Wellington

6. Start with a backbone • The vertical dimension of IL • Outcomes logic, not processes or activities • Outcome grammar • Connectedness • The necessary but not sufficient rule • Advisor’s mindset • Optimistic • Skeptical Victoria University of Wellington

7. Political benefits of simplicity Analytical pitfalls What’s wrong with this backbone? Can the matrix fill in the gaps? The “If you build it, they will come” backbone Reduce traffic congestion End outcome People drive on it Immediate impact Build bypass Output Victoria University of Wellington

8. The more complex backbone • Add intermediate outcomes • More assumptions about indirect causation Victoria University of Wellington

9. Ultimate outcome: Increased educational achievement/smarter kids Some “bad” schools lift their game “Good” schools get even better Some “bad” schools fold New “good” schools come on line “Good” schools gain pupils “Bad” schools lose pupils Parents choose “best” schools for their children Parents possess “appropriate” information about schools Parents aware of & understand program Output: School vouchers Victoria University of Wellington

10. Social policy: Note the large leaps in logic that often occur at the top The black box at the top of the backbone Ultimate outcome realised Client’s behaviour changes (How? Why?) Client responds well to services Victoria University of Wellington

11. Plot twists • When is an intermediate outcome also an end outcome? • Can one agency’s / department’s intermediate outcome be another agency’s end outcome? Victoria University of Wellington

12. The backbone as a manage-ment tool: Links below outputs Source: R Waite Victoria University of Wellington

13. The backbone as a risk ID tool: Collateral outcomes Victoria University of Wellington

14. What are IL’s prerequisites? • Agreed outcomes for the top row • Sources • Statement of intent • Agency/departmental mission • The importance of first principles review • The role of problem definition • Outcomes (goals) are the flipside of problems • The “problem logic” model and the black box • Intervention option(s) for the bottom row • Common sense Victoria University of Wellington

15. Group Exercise 1 • Work in pairs • Choose a familiar policy/output from your work or from the news • Produce a backbone linking the output to intermediate and ultimate outcomes • Identify strong and weak links Victoria University of Wellington

16. Move to a matrix (Funnell 1997)

17. Conventional uses Testing existing policy hypotheses Testing performance Improving impacts through design & management of risk Making better use of existing data Unconventional uses Comparing policy options Identifying generic intervention templates for a department Discovering/ inventing new interventions What are its uses? Victoria University of Wellington

18. Testing existing policy hypotheses • If X, then Y • Y = f (X) • Does the raw logic hold? (ex ante) • Does the available evidence support the logic? (ex ante and ex post) • What additional evidence is needed to test the logic? • IL breaks an impact evaluation into chunks. Victoria University of Wellington

19. Testing/confirming performance Column 6 in the IL matrix allows us to • disaggregate performance into chunks • distinguish chunks that are working well from those working less well • based on achievements compared agains success criteria/targets Victoria University of Wellington

20. Improving impacts • Identify conceptual and operational gaps in existing policy • Target issues for review (weak links) • Monitor • Internal and external risks • Counter-intuitive causes and effects • Revise design Victoria University of Wellington

21. Making better use of data • Evidence need not relate to ultimate/end outcomes to be useful • Findings to date (from NZ or international) may shed light on immediate and intermediate links in the chain • Role of research in the “problem logic” • Examples • School choice research and its place in the IL Victoria University of Wellington

22. Comparing policy options (via the conventional matrix) Multiple outcomes = unlinked “criteria” Victoria University of Wellington

23. Compare #’s of links More links = more chances to stuff it up/more resources required? More links = less uncertainty, more robust theory? Fewer links = political plus? Compare #’s and magnitude of weak links Compare #’s and magnitude of possible unintended outcomes Using IL to compare options Step 1: Prepare a backbone for each option Victoria University of Wellington

24. Using IL to compare options Step 2: Prepare an IL matrix for each option Victoria University of Wellington

25. See next slide Compare risks across A, B, C Compare resources needed A, B, C Compare performance contract possibilities Compare evaluability IL sets up more accurate cost-effectiveness analysis Remove unnecessary steps before costing Identify possible sources of extra costs Using IL to compare options Step 3: Compare across IL matrices Victoria University of Wellington

26. A cross-cutting matrix *Numbers in parentheses refer to columns in the IL matrix Victoria University of Wellington

27. Identifying IL templates • The case management model • Generic steps (slide 28) • Early intervention example • The information campaign model • Generic steps (slide 29) • The deterrence model • Mandatory sentencing laws example (slide 30) • The pollution permits model • GHGs example (slide 31) Victoria University of Wellington

28. Reduced long-term costs and/or increased long-term benefits to the community Life circumstances/chances of individual are improved; long-term objectives are achieved Short-term objectives for individual progressively achieved Individualised programme put in place to meet objectives Realistic objectives set for (and with) the individual Individual’s needs & prospects assessed accurately Output: Target group enters programme Victoria University of Wellington

29. Behaviour change leads to improved outcomes Readers influence others to change opinions/behaviour Readers change their behaviour Readers change their opinions Readers learn the facts Audience reads literature Appropriate audience receives literature Literature passes pretest for readability, etc. Output: Educational literature produced Victoria University of Wellington

30. Rates of crime X X offenders work harder to avoid apprehension Potential offenders avoid crime X Fewer X offenders on the street Past & potential offenders include new X sentencing risk in their personal decision making More X offenders jailed longer Past & potential offenders aware of sentencing Judges understand and apply them Output: Mandatory prison sentences for crime X Victoria University of Wellington

31. GHGs & costs Innovations in clean technology diffuse Regulators sanction “Clean” firms profit Some plants buy add’l permits Some plants emit GHGs above permit Some plants invest in clean R & D and technology Plants calculate costs & benefits of investing in cleaner technology v buying add’l permits v paying fines for excessive GHGs Govt invests auction revenue in clean R&D Permits auctioned to bidders (or other allocation made) Output: Tradable emissions permits created for GHGs Victoria University of Wellington

32. Discovering/inventing new interventions • The brainstorming approach • Pick generic policy instruments • Apply to the problem at hand, using quick, back-of-the-envelope backbones • The engineering approach • Start with the “problem logic” • Find the entry points in the model • Fashion interventions for the entry points Victoria University of Wellington

33. Group exercise 2 • Same pairs • Choose an end/ultimate outcome and make it the top “vertebra” of a backbone • Work down to identify intermediate outcomes that might lead to that end outcome (based on your knowledge of how that outcome is “naturally” produced) • What interventions suggest themselves as you move down? Victoria University of Wellington

34. What are IL’s blind spots? Equity • Might the chain of outcomes look different for different population groups of interest? • Might risk factors differ across groups? • Might different groups need different activities and resources to reach each intermediate outcome? Victoria University of Wellington

35. What are IL’s blind spots? • Hidden portions of the backbone • Inputs and activities (below) • Collateral outcomes (beside) • Program/theory assumptions (beside) • Getting trapped in a paradigm • Tikanga v cognitive-behavioural paradigms for explaining crime • Focusing on the lower levels of the hierarchy, where managers have more control Victoria University of Wellington

36. How do we minimise the blind spots? • Research that contributes to robust problem logics • The poverty example • The drug harms example • Evaluation that contributes to robust intervention logics • The welfare to work example • Outcomes that reflect actual results in the community/real consequences Victoria University of Wellington

37. How do we know when an IL is working (or not)? • Does it help us distinguish between apparently more and less promising interventions? • If it just rationalises everything, not robust • Does it systematically favour some types of interventions over others? Why? Is this warranted (cross check)? • Does it help us make better use of existing evidence? • Does it help us generate a research agenda? Victoria University of Wellington

38. Ex ante criteria for a good IL • Proper “grammar” in the backbone • Outcomes, not processes or activities • Each intermediate outcome represents a necessary but not sufficient cause of the next outcome • Success criteria are measurable and lend themselves to targets • Activities and resources cover all of the key factors within the programme’s control • Activities and resources supply what is needed to get from one outcome to the next Victoria University of Wellington

39. Ex post criteria for a good IL • Are outcomes being produced more cost effectively than prior to use of IL? • Are intended and unintended outcomes predicted more accurately? • Are there fewer unintended outcomes? • Are there fewer unexpected outcomes/surprises? • Is the department accumulating better information about its own performance? Victoria University of Wellington

40. Through multiple applications, learning by doing Through cross-breeding with other soft & hard systems approaches Through peer review Challenges to be met Accounting for new sets of surprises Facilitating cross-departmental thinking on partnerships for particular outcomes Facilitating equity analysis Other How can IL evolve? Victoria University of Wellington