Evaluating impact of OVC programs: standardizing our methods

1. Evaluating impact of OVC programs: standardizing our methods Jenifer Chapman, PhD Futures Group/MEASURE Evaluation

2. Overview • Evaluation vs. monitoring vs. case management • Update on the MEASURE Evaluation OVC program evaluation tool • Key questions

3. Evaluation vs. monitoring vs. case management Case management questions • What are the child’s immediate priority needs? • Is child in school now? • Have earlier priority needs abated? Monitoring questions • How many children are receiving services? • How many social workers have been trained? • Is the program being implemented as planned?

4. Evaluation vs. monitoring vs. case management Evaluation questions: • Did program beneficiaries show improved well-being over time? • What proportion of households are food secure, compared to 5 years ago? • Can improvements in well-being be attributed to the program?

5. Tools for answering these questions • We have case management tools, and monitoring tools… • We lack a standardized approach and tool for evaluating the impact of OVC programs globally

6. An evaluation tool, in context • Purpose: To assess the impact of a program (on child/household wellbeing) over time • Who uses it?: Trained data collectors / researchers • Among whom is it applied? • A sample of beneficiaries • Sometimes a matched group of potential beneficiaries

7. What is an evaluation tool? • What information is collected? • Client contact information / demographics • Wellbeing information that is changeable over time • Services provided • Information must be useful at program/national level (different information than needed for case management/care planning)…

8. Evaluation protocols • Evaluation requires a protocol detailing: • Sampling and recruitment strategy • Data collection procedures • Data management and data analysis • Dissemination • Ethical approval is required. Evaluation is not part of routine QI involving service providers.

9. MEASURE Evaluation OVC Program evaluation toolkit

10. The problem Lack of standardized, objectively verifiable, impact indicators of child wellbeing

11. The vision • Quantitative child outcomes and household outcomes measurement tools, valid across countries at the population level • With survey protocol, analysis plan and report template, and training materials • Progress: • final shortlist of 12 child & 3 caregiver/HH impact indicators • Draft tools in discussion • Piloting in Nigeria, Mozambique, Zambia & Ethiopia

12. Methods • Extensive research (lit reviews, international/national child wellbeing tools and indicators, OVC program evaluation tools, national OVC M&E plans, DHS, etc.)  Result: 100s of child & household wellbeing indicators • Indicator analysis against defined criteria & wide stakeholder discussions  Result: shortlist of indicators

13. Inclusion criteria • Measures impact/outcomes • Amenable to change from program interventions • Relevant across a wide range of interventions • Contributes to a holistic vision of child wellbeing • Objectively verifiable • Easy to implement • Relevant across different regions / countries • Relevant or easily adapted across age and sex

14. 12 Child wellbeing indicators • Percent of children malnourished • 6-59 months old: measurement of middle-upper arm circumference (MUAC) • >60 months old: measurement of weight and height • Percent of children with recent diarrhea • Have you / has the child had diarrhea in the two weeks preceding the survey?

15. 12 Child wellbeing indicators • Percent of children with recent fever • Have you / has the child had a fever in the two weeks preceding the survey? • Percent of children who are too sick to participate in daily activities • In the last 2 weeks, have you / has the child been too sick to participate in daily activities?

16. 12 Child wellbeing indicators • Percent of children reporting irregular food intake • Have you / has the child gone a whole day or night without eating in the last 4 weeks? • Percent of children fully immunized • Has child received all age-appropriate immunizations?

17. 12 Child wellbeing indicators • Percent of children with basic shelter • Is the place that you / the child slept last night protected from the weather? • Percent of children with basic social support • Is there someone you / the child can go to, to help solve a problem?

18. 12 Child wellbeing indicators • Percent of children who have a birth certificate / identification card • Does the child have a birth certificate or registration / ID card? • Percent of children currently enrolled in school • Are you / is the child currently enrolled in school?

19. 12 Child wellbeing indicators • Percent of children regularly attending school • During the last school week, did you / the child miss any school days for any reason? • Percent of children who progressed in school over time • What is the highest level of education you have / the child has completed?

20. 3 Household wellbeing indicators • Percent of households in which caregiver reports basic social support • Is there someone you can go to, to help solve a problem? • Percent of households that are food insecure due to lack of resources • In the past 4 weeks, was there ever no food to eat of any kind in your household because of a lack of resources to get food?

21. 3 Household wellbeing indicators • Percent of households able to access money to meet important family needs • If you needed money to meet an important family need, such as to pay for school fees, pay for transportation, or purchase for food, how would you pay? (smaller expenses) • If you needed money to meet an important family need, such as to pay for a family emergency, pay for a house repair, or pay for medical treatment, how would you pay? (larger expenses)

22. And the tool? • Indicators form the basis of two tools: • Child questionnaire (ages 10-17) • Caregiver questionnaire (including questions on caregiver wellbeing, household, child wellbeing) • Draft tools available Autumn 2012 • Tools will be piloted in Autumn/Winter 2012 • Final tools & guidance documents available mid-2013

23. For discussion • PSS and HES indicators – your experience, please! • Methodological fine points (or not so fine points)…what works, what does not? • How is your organization currently evaluating programs? • How will this tool support you in evaluation? • What more is needed?

24. The research presented here has been supported by the President’s Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of MEASURE Evaluation cooperative agreement GHA-A-00-08-00003-00. Views expressed are not necessarily those of PEPFAR, USAID or the United States government. MEASURE Evaluation is implemented by the Carolina Population Center at the University of North Carolina at Chapel Hill in partnership with Futures Group, ICF International, John Snow, Inc., Management Sciences for Health, and Tulane University.

25. Round-up

26. Our Community of Practice • There is lots going on in OVC M&E but cross-country dialogue is lacking • ChildStatusNet is being revamped as an OVC M&E Community of Practice (or even M&E of community-based interventions) • Forum to exchange ideas & share tools / indicators / methods • Regular webinars with guest speakers

28. For more information, contact: Jenifer Chapman: jchapman@futuresgroup.com Karen Foreit: kforeit@futuresgroup.com http://www.cpc.unc.edu/measure/