Really Using (Useful) Theories of Change

1. Really Using(Useful) Theories of Change IPDET 2013 John Mayne, Advisor on Public Sector Performance john.mayne@rogers.com

2. Using Theories of Change • Understanding and agreeing on interventions • Addressing equity issues • Designing interventions • Ex ante evaluation • Designing monitoring systems • Managing adaptively • Reporting performance • Designing evaluation questions, methods and tools • Making causal claims • Generalizing to other locations

3. A Generic Results Chain Impacts Direct Benefits Behaviour changes External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Activities and Outputs with respect to beneficiaries Timeline

4. A Generic Results Chain Impacts • Note • No labeling of outcomes types • Focus on program theory • Timeline • Built on capacity and practices changes • Explicit recognition of reach • Feedback loops Direct Benefits Behaviour changes External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction Activities and Outputs with respect to beneficiaries Timeline

5. Assumptions and Risks Impacts Direct Benefits Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Peer or trend pressure; other interventions Behaviour changes External Influences Capacity changes in knowledge, attitudes skills, opportunities & incentives Reach & Reaction An assumption is NOT a description of the causal link Activities and Outputs with respect to beneficiaries Timeline

6. A Generic Beneficiaries Theory of Change Impacts Impact Assumptions& Risks Other explanatory factors Direct Benefits Direct Benefits Assumptions& Risks Other explanatory factors Behaviour changes External Influences Behaviour Change Assumptions& Risks Other explanatory factors Capacity changes in knowledge, attitudes skills, opportunities & incentives Capacity Change Assumptions& Risks Other explanatory factors Reach & Reaction Reach Assumptions& Risks Activities and Outputs with respect to beneficiaries Timeline

7. Results Chain for Enhancing Education Outcomes for Girls Improved education outcomes for girls Girls learning improves Intervention to improve educational outcomes for girls by providing gender sensitivity training to teachers, in a region where girls education has not been a priority. Girls more engaged in and wanting an education Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Awareness and skills training provided to teachers on girls’ education needs timeline

8. Other Factors for Enhancing Education Outcomes for Girls Girls learning improves Girls more engaged in and wanting an education • Access to schools • Accommodation in schools • Parents and community support • Time for studying Teachers provide girls with more empathetic and supportive teaching in schools

9. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls • Girls’ Direct Benefit Assumptions& Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Girls learning improves Unintended effect: Boys’ education suffers Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources Capacity changes in teachers’ knowledge, attitudes and skills All relevant teachers get the training Awareness and skills training provided to teachers on girls’ education needs timeline

10. Other factors Seems clear that more than training teachers may be required, such as: • Getting girls to school • Parental and community support • Schools accommodate girls • Teachers support girls education • Empathetic teaching is rewarded • Focus on and resources for girls’ education are ongoing (outcomes take time)

11. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Girls learning improves Unintended effect: Boys’ education suffers • Girls’ Direct Benefit Assumptions& Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources • Teacher Behaviour Change Assumptions& Risks • Training relates to the local conditions • Teachers are assessed in part on their application of the training Capacity changes in teachers’ knowledge, attitudes and skills • Capacity Change Assumptions and Risks • Most teachers want to better education for girls All relevant teachers get the training • Reach Assumptions& Risks • Only teachers with a predisposition to the importance of girls’ education attend Awareness and skills training provided to teachers on girls’ education needs timeline

12. Equity: Distribution of impacts • As set out so far, ‘girls’ is an general group • But among ‘girls’, there are worst-off girls, who may need special attention (e.g., access, income, conservative parents) • And as already noted, there may be unintended impacts on boys • To get at these distributional affects, we can build nested theories of reach: • A ToC for a particular reach group within an overall ToC • And can then tell reach impact stories about the effects and impacts of different reach groups

13. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls Worst-Off GirlsTheory of Reach • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Girls learning improves Unintended effect: Boys’ education suffers Girls Theory of Reach • Girls’ Direct Benefit Assumptions& Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources • Teacher Behaviour Change Assumptions& Risks • Training relates to the local conditions • Teachers are assessed in part on their application of the training Capacity changes in teachers’ knowledge, attitudes and skills • Capacity Change Assumptions and Risks • Most teachers want to better education for girls Teachers Theory of Reach All relevant teachers get the training • Reach Assumptions& Risks • Only teachers with a predisposition to the importance of girls’ education attend Awareness and skills training provided to teachers on girls’ education needs timeline

14. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Girls learning improves Unintended effect: Boys’ education suffers • Girls’ Direct Benefit Assumptions& Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources • Teacher Behaviour Change Assumptions& Risks • Training relates to the local conditions • Teachers are assessed in part on their application of the training Capacity changes in teachers’ knowledge, attitudes and skills • Capacity Change Assumptions and Risks • Most teachers want to better education for girls All relevant teachers get the training • Reach Assumptions& Risks • Only teachers with a predisposition to the importance of girls’ education attend Awareness and skills training provided to teachers on girls’ education needs timeline

15. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Girls learning improves Unintended effect: Boys’ education suffers • Girls’ Direct Benefit Assumptions/Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources Supporting activities • Teacher Behaviour Change Assumptions& Risks • Training relates to the local conditions • Teachers are assessed in part on their application of the training Capacity changes in teachers’ knowledge, attitudes and skills • Capacity Change Assumptions and Risks • Most teachers want to better education for girls All relevant teachers get the training • Reach Assumptions& Risks • Only teachers with a predisposition to the importance of girls’ education attend Awareness and skills training provided to teachers on girls’ education needs timeline

16. Theory of Change for Enhancing Education Outcomes for Girls Improved education outcomes for girls • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Girls learning improves Unintended effect: Boys’ education suffers Girls Theory of Reach • Girls’ Direct Benefit Assumptions/Risks • Girls have time to study • Girls education supported by parents and community • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Parents Theory of Reach Girls more engaged in and wanting an education • Girls’ Behaviour Assumptions& Risks • Girls have access to schools • Schools can adequately accommodate girls • Other explanatory factors • Internet access becomes widely available Teachers provide girls with more empathetic and supportive teaching in schools External Influences Parents Peers Religion Culture Education resources Supporting activities • Teacher Behaviour Change Assumptions& Risks • Training relates to the local conditions • Teachers are assessed in part on their application of the training Capacity changes in teachers’ knowledge, attitudes and skills Teachers Theory of Reach • Capacity Change Assumptions and Risks • Most teachers want to better education for girls All relevant teachers get the training • Reach Assumptions& Risks • Only teachers with a predisposition to the importance of girls’ education attend Awareness and skills training provided to teachers on girls’ education needs timeline

17. Theory of Reach for Enhancing Education Outcomes for Worst-Off Girls Improved education outcomes for worst-off girls timeline • Impact Assumptions& Risks • Resources and will continue for multi-year involvement Worst-Off girls learning improves Unintended effect: Boys’ education suffers Support Activities • Worst-Off Girls’ Behaviour & Direct Benefit Change Assumptions& Risks • Worst-off girls have access to schools • Worst-off girls education supported by parents and community • Worst-off girls have time to study • Schools can adequately accommodate girls • Focus on girls is not disruptive to boy’s education • Other explanatory factors • Internet access becomes widely available Partner Activities Nested theory of reach for access by worst-off girls Worst-off girls are provided transportation to schools Engagement with governments Worst-Off girls more engaged in and wanting an education Engaged with parents & communities with worst-off girls Engagement with NGOs Teachers provide (all) girls with more empathetic and supportive teaching in schools intervention activities   timeline results

18. A Basic Theory of Change for Comprehensive Interventions Timeline Impact  Supporting Activities and Outputs Reach, Capacity change Behavioural Change Impact Assumptions & Risks Other Explanatory Factors Direct Benefit Changes Direct Benefits Assumptions & Risks Other Explanatory Factors Behaviour Changes External Influences Behavioural Change Assumptions & Risks Other Explanatory Factors Capacity Changes   Capacity Change Assumptions & Risks Other Explanatory Factors Reach & Reaction  Reach Assumptions & Risks Activities and Outputs Timeline

19. A Basic Theory of Change for Comprehensive Interventions Timeline Impact  Supporting Activities and Outputs Reach, Capacity change Behavioural Change Different Beneficiary Nested ToC Impact Assumptions & Risks Other Explanatory Factors Direct Benefit Changes Direct Benefits Assumptions & Risks Other Explanatory Factors Behaviour Changes External Influences Behavioural Change Assumptions & Risks Other Explanatory Factors Different Intermediary Nested ToC Capacity Changes   Capacity Change Assumptions & Risks Other Explanatory Factors Reach & Reaction  Reach Assumptions & Risks Activities and Outputs Timeline

20. Using Theories of Change • Designing monitoring systems • likely need more than just a focus on results—the whole ToC, including assumptions and risks • Adaptive management • Identify what to be monitored • On a regular basis, reflect on what is being observed and on your ToC • Revise your intervention activities and your ToC as you learn

21. Using Theories of Change • Reporting performance • A verified ToC is your performance story • Designing evaluation questions, methods, tools • Identifies questions • Adds power to case studies by confirming the ToC • Basis for surveys et al

22. There are potential Evaluation Questions associated with each box and arrow Impacts Direct Benefits Behaviour Change Assumptions: How are changes in knowledge, attitudes, skills, opportunities and/or incentives expected to change behaviour? What events and conditions have to happen? What factors influence these processes? Risks: Risks to the link not occurring. Other Explanatory Factors: Peer or trend pressure; other interventions Behaviour changes External Influences • Behaviour Change • What behavioural (practice) changes are observed? • Causal Question • To what extent are the behavioural changes the result of the capacity changes? • Assumption Questions • Did the events/conditions occur? • Did risks materialize? • Are other explanations plausible? Capacity changes in knowledge, attitudes skills, opportunities & incentives • Capacity Change • What capacity) changes are observed? Reach & Reaction Activities and Outputs with respect to beneficiaries Timeline

23. Using Theories of Change • Making causal claims • Based on generative causality • ToC are models of causality • Approaches such as contribution analysis demonstrates causality

24. Theories of Change asCausal Packages ToC is a model of the intervention as a contributing cause • ToC identifies supporting factors (assumptions) and confounding factors (risks) • Intervention activities plus the assumptions are sufficient to bring about the intended impacts • Explains how and why intended impacts are expected to occur

25. A Basic Theory of Change Timeline Impact  Supporting Activities and Outputs Impact Assumptions & Risks Other Explanatory Factors Direct Benefit Changes Direct Benefits Assumptions & Risks Other Explanatory Factors Behaviour Changes External Influences Behavioural Change Assumptions & Risks Other Explanatory Factors Capacity Changes   Capacity Change Assumptions & Risks Other Explanatory Factors Reach & Reaction  Reach Assumptions & Risks Activities and Outputs Timeline Causal package

26. Contribution Analysis Using a generative perspective on causality, CA shows that an intervention is a contributory cause: • The expected result occurred • The ToC (causal package) is sufficient • supporting factors (assumptions) occurred and any other supporting factors have been included in the ToC • plausible rival explanations have been accounted for • The intervention is necessary for the ToC to be sufficient • And can explore the role the intervention played, such as a trigger

27. Using Theories of Change • Generalizing • What works where and when? • ToC explicitly includes the context and the causal mechanisms • Confirming a ToC provides significant evidence on if and how the intervention would work elsewhere

28. Messages • ToC are worth the effort to develop • Not a throw away, but a powerful tool to use in evaluation and in managing • Lots of fun … and lots of debate!

29. References Mayne, J. (forthcoming). Using Theories of Reach to Enhance Equity Considerations in Evaluation. In Evaluation and Equity. K. Forss and M. Marra, Eds: Transaction. Mayne, J. (2013). Making Causal Claims. Connections, June. Available at http://www.europeanevaluation.org/images/file/files/ees-newsletter-2013-06-june.pdf. Mayne, J. (2012). Contribution Analysis: Coming of Age? Evaluation 18(3), 270-280. Mayne, J. (2012). Making Causal Claims, ILAC Brief No. 26: The Institutional Learning and Change Initiative. Available at http://www.cgiar-ilac.org/files/publications/mayne_making_causal_claims_ilac_brief_26.pdf Stern, E, Stame, N., Mayne, J., Forss, K, Davies, R. and Befani, B. (2012). Broadening the Range of Designs and Methods for Impact Evaluation, Working Paper 38, Department for International Development, London. Available at http://www.dfid.gov.uk/R4D/Output/189575/Default.aspx Mayne, J. (2008). Contribution Analysis: An Approach to Exploring Cause and Effect, ILAC Brief 16. Available at http://www.cgiar-ilac.org/files/publications/briefs/ILAC_Brief16_Contribution_Analysis.pdf