Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012

1. Africa RISING M&E Expert Meeting Addis Ababa, 5-7 September 2012 Monitoring and Evaluation (M&E) strategy

2. Outline • Timeline • Objectives of M&E • Targeting vs Research • Monitoring and Evaluation • Theory of Change • Evaluation designs • Open issues

3. Timeline Component Inventory Site selection/ Characterization Component DB Activity -> Indicator List Evaluation Design Survey Design Baseline Survey ✔ ✔ 1-3 Months 3-9 Months 9-12 Months

4. Objectives of M&E • Keep track of the AR’s outputs (M) • Assess the impact of AR (E) • Create knowledge of what works (M&E): -assess effectiveness -rank policy/project alternatives -appraise scale-up and external validity -identify problems in the project -inform project management

5. What is AR M&E after? • M could be an easy(ier) task but: -we need constant and timely information -we need close collaboration among CG, implementers, and IFPRI ->Solution: Outcome mapping? Cost-benefit analysis? Surveys? • E must be well thought in advance…

6. Targeting vs Research 1. Do we need to select the poorest/hungriest farmers to benefit from AR? 2. Or, being AR a research project, is it possible to use a stratified sample (based on different criteria)? 2a. common E approach? 2b. project-specific E approach? It depends on the question(s)!

7. Evaluation • Determine the causal effect of AR on outcomes (not only on outputs): • Farmers’ wellbeing? • Land productivity? • Input supply, labor productivity, environment, women’s conditions, health and nutrition,…? • …all of the above plus-> for whom? For which development domain? For which type of households? For which livelihood? • What would be the impact with a different technology package?

8. BEHAVIOUR MONITOR EFFICIENCY INPUTS OUTPUTS OUTCOMES/IMPACTS EVALUATE EFFECTIVENESS $$$ Monitoring Evaluation ‘Traditional’ M&E: monitoring to track implementation efficiency (input - output) Impact Evaluation:estimate causal effectiveness on outcomes (output - outcome) Note: Diagram from WorldBank training material produced by Arianna Legovini, Lead Economist - AIEI

9. Theory of Change • Impact evaluation must be based on a set of hypotheses on the change that can be achieved as a consequence of AR • How would you think that AR can affect the life of (beneficiary) farmers?

10. AR implementation ACTIVITIES Targeting OUTPUTS Productivity increase FIRST ORDER OUTCOMES Saving/ Investment Time Allocation/ Caring arrangements/ Migration of Adults and Caregivers * Labour Participation Child Labour Income and Expenditure Intra-household decision making * Expenditure on Health and Education for children Other Expenditures for children: Food, Clothing, Recreation General Household Expenditure SECOND ORDER OUTCOMES Asset Building Time and risk preferences THIRD ORDER OUTCOMES Utilization of Health Services Livelihood Strategies /Coping Strategies /Vulnerability to Shocks Health Status Time Allocation of Children School Enrolment and Attendance Food Intake Dietary Diversity Food Security Remittances Access to Credit Psychological well being * School Progression

11. Impact Evaluation • How would you go about measuring the causal impact of AR on … -productivity?

12. Impact Evaluation - Method Y Bens Impact? t Before After

13. Impact Evaluation - Method Impact Y Bens Non Bens RCTs t Before After

14. Impact Evaluation - Method Diff in Diff Impact Y Pre-existing Difference Bens Non Bens t Before After

15. Impact Evaluation - Method Diff in Diff Impact Y with Propensity Score Matching Bens Non Bens t Before After

16. Impact Evaluation • How would you go about measuring the causal impact of AR on … -Gross margin per unit of land? [assume we are not interested in farmers with <.3 ha]

17. RDD

18. Impact Evaluation - Method • Causal effect: change that is due to AR and not to other actors or factors (confounders) • … taking into account any other factors also changing during the program period • … taking into account any systematic differences between beneficiaries and non-beneficiaries of AR intervention It is very important that the “control group” is comparable to the “treatment group”

19. Evaluation Design Treatment Sites Control Sites • How can we ensure that treatment and control sites are comparable?

20. Evaluation Design • How can we ensure that treatment and control farmers are comparable?

21. Evaluation Design Random Treatment Assignment

22. Evaluation Design Random Treatment Assignment

23. Evaluation Design Control Sites Treatment sites BENEFICIARY FARMERS WOULD BE BENEFICIARY FARMERS B A

24. Where do we stand? BASELINE Impact Y A Pre-existing Difference B t Before After

25. Evaluation Design Treatment sites Control sites BENEFICIARY FARMERS WOULD BE BENEFICIARY FARMERS A B NON BENEFICIARY FARMERS WOULD BE NON BENEFICIARY FARMERS C D

26. Open issues/1 What questions we would like AR to answer? What R4D lessons can we learn from jumpstart projects? What would you carry over to longer-term AR activities? Do we need to target AR to specific farmers? Role of FtF indicators -which data to collect-?

27. Open issues/2 Choice of outcome indicators/variables Ethics for control (same as placebo effect in medicine) Sampling frame for randomization Sample design Statistical power for causal impact Site 1 Site 2