Impact Measurement why what how

1. Impact Measurementwhywhathow 9-14-09 Atlanta

2. Today Imperatives Questions Why Now? Significant Challenges Breakthroughs in the field CARE’s Long-Term Strategy This Year How do we meet the challenges?

3. Imperatives Measurement … …systems must actually add value for the organization and produce reliable data that it can be utilized to inform strategic decisions. …can only flow from a clearly-defined program approach that is predicated on a clear framing of the problem.

4. Questions • If you do not measure our impact and demonstrate the change, then does it matter? • Can we measure the most important changes or do we emphasis what we can measure? • How do you actually measure social impact and do those type of changes inherently defy measurement - an increase in someone’s level of opportunity or their dignity • Does spending on measurement and building measurement systems mean less time and resources for the actual work that we do?

5. Most Significant Challenges What does ‘impact’ even mean? How do we even begin to define metrics that can cut across a plethora of disparate issues that CARE is addressing? Do we need counter-factual, experimental designs and benchmarks within the same issue areas to do so? How do you measure outcomes rather than inputs? Especially when the intervention is removed from the ultimate desired change - i.e., girl’s education programs with the goal of increasing women’s empowerment.

6. Most Significant Challenges (2) How do we measure impact for the wide variety of constituents that CARE is accountable to? How do we build the capacity across the organization to make sure that the measurement is consistent through a well implemented system.

7. Breakthroughs in the Field Shared Measurement Platforms Allow organizations to choose from a set of measures within their fields Comparative Performance Systems Require all participants within a field to report on the same measures Adaptive Learning Systems Allow people working on different aspects of a single complex issue to engage in an ongoing, facilitated process

8. Why Now ? • The program approach across CARE • Critical to demonstrate intended benefits • for organizational performance management • Support fundraising • Provide evidence for advocacy for LIFT-UP • Proactively meets future compliance

9. Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle: Impact Effects Outputs Activities Inputs

10. Levels of Measurement The proposed system covers all levels of measurement and all stages of the program cycle: Impact Effects Outputs Activities Inputs

11. Levels of Measurement • LEVEL 1 • Input: • Resources needed to undertake a set of activities • Activity: • The actions that convert inputs into outputs • Output: • The changes that are a direct result of activities

12. Levels of Measurement • LEVEL 2 Effect: • Changes in human behaviors and practices • Improvement in services through increased competency (e.g. more effective extension systems) • Changes in systems and policies (enabling environment) • LEVEL 3 Impact: • Long-term sustainable change • Contribution to the MDGs

14. Components Component 1 Program Data Collection System Sectoral Portfolio Review Component2 Annual Impact Report Strategic Impact Inquiry Component 3 Meta-Evaluation Proposal Analysis Program Causal Analysis

15. Impact Effects Components of IM System Inputs Activities Outputs

16. Access AfricaHow are we measuring our impact

17. Impact Measurement System Impact Level % population below $2 per day (MDG) Increased general well being of participants household Outcome Level % of VS&L members who have invested in productive assets % women holding leadership position at community level Level of implementation of policies for women right on land and other productive assets Increased in economic, social and political empowerment participants Activity/Output Level R2. Retention rate R5. Annualized return on savings R8. Portfolio at risk R16. Cost per member assisted Monitoring & VSLA quality

18. Activity & Output level: MIS System • Excel based system to Manage Field Officer performance & VSLA Quality • Measures 4 categories of performance ratios (SEEP): • Members Satisfaction • VSLA Financial Performance • VSLA operating efficiency • CARE operating efficiency • Quarterly data collection by projects in the field • Access Africa aggregates global data quarterly • Data enables AA to engage in Program Quality discussions with the COs • Common system used by OXFAM, CRS, PLAN, Aga Khan, FfH, etc.

19. What about Outcome and Impact? • Common indicators/tool across COs • Option of methodology: • Client Survey: data collection on participants household • Household Survey: data collection on community members (participants and non-participants) • RCT, with IPA in Uganda and Malawi • Mixing quantitative and qualitative methods • Capacity building of COs – TDY, direct support from AA • Support COs in hiring relevant staff and/or consultants for M&E

20. Making the information Available

24. The Directory Name Program # Goal Manager Contact Information Associated Project Name PN/FC # Manager Contact Information Name PN/FC # Manager Contact Information Program Information Project Information Name PN/FC # Manager Contact Information Name PN/FC # Manager Contact Information • Searchable information in both categories: • Start Dates and End Dates of Projects and Programs • Impact Population • Target Group • Signature Program component • MDG Impact Area program or project works in • Number of direct participants in the program • Gender and Age • Activities and Outputs • CARE partners • Advocacy Goals, Partners and Targets of Advocacy Efforts

25. This year • Work with Country Offices to create and disseminate appropriate guidance on impact measurement • Work with Technical Units in Atlanta to help put impact measurement systems in place in select Learning Labs. • Focus on organizing consultations on and outlining a business process for impact measurement at CARE, complete with a resource development plan • Working with the Policy Advocacy Unit and Communications on CARE’s Annual Impact Report.

26. Finished Finally