1 / 29

PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011)

PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011). Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author) , Natasha Croome , Adele Daniels and Jaya Karira. Task: Summarize and Analyze PEPFAR OVC Evaluations. 18 Evaluations 22 Programs

tamah
Download Presentation

PEPFAR OVC Evaluation: How good at doing good? (GH TECH/USAID 2011)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. PEPFAR OVC Evaluation: How good at doing good? • (GH TECH/USAID 2011) Prof Lorraine Sherr UCL London UK Acknowledgements to Miriam Zoll (Co-author), Natasha Croome, Adele Daniels and Jaya Karira

  2. Task: Summarize and Analyze PEPFAR OVC Evaluations 18 Evaluations 22 Programs 9 Countries – Kenya, Tanzania, Uganda, Mozambique, Rwanda, Namibia, Zambia, South Africa and Haiti 8 basic needs categories: Food/Nutrition, Health, Education, PSS, Economic Strengthening, Child Protection, Shelter and HIV Prevention Critique methodology and summarised findings. http://resources.ghtechproject.net/content/pepfar-ovc-evaluation-how-good-doing-good

  3. Evaluation – is it ever expedient? 1. Systematic reading by reviewers. 2. Coding - extract themes agreed by two reviewers. 3. Sub-analysis of costing. 4. Methodology adequacy criteria. 5. Seven basic needs categories. 6. Recommendations for future evaluation and evidence- based policy. Double coding 6 reviews (Kappa =.83)

  4. Evaluations Surveyed Brewster, Lee, et al, Catholic Relief Servies' OVC PEPFAR Programs: A Midterm Multicountry Program Evaluation Nyangara, F., Obiero, W., Kalungwa, Z., Thurman, T., Community-based Psychological Intervention for HIV-Affected Children and their Caregivers: Evaluation of the Salvation Army's Mama Mkubwa Program in Tanzania, USAID, AXIOS, Measure Evaluation, March 2009 Thurman, T., Hutchinson, P., Ikamari, L., Gichuhi, W., Murungaru, K., Nyangara, Community Education and Sensitization as an OVC Care and Support Strategy: Evaluation of the Integrated AIDS Program-Thika in Kenya, USAID, University of Nairobi Population Studies and Research Institute, Measure Evaluation, March 2009. Thurman, T., Rice, J., Ikamari, L., Jarabi, B., Mutuku, A., Nyangara, F., An Evaluation: the Difference Interventions for Guardians Can Make: Evaluation of the Kilifi Orphans and Vulnerable Children Project in Kenya, USAID, Measure Evaluation, University of Nairobi Population Studies and Research Institute, March 2009. Nyangara, F., Kalungwa, Z., Obiero, W., Thurman, T., Chapman, J., Promoting a Home-based Program Model for Supporting Children Affected by HIV/AIDS: Evaluation of Tumaini Project in Iringa Region, Tanzania, USAID, AXIOS, Measure Evaluation, March 2009

  5. Evaluations Surveyed 6. Swarts, Brian, et al, Empowering Better Care: Report on Economic Strengthening for OVC Caregivers in Uganda, Summary of Findings, The Salvation Army World Service Office, WORTH, USAID and PACT, April 2010 7. Hutchinson, P., Thurman, T., Analyzing the Cost Effectiveness of Interventions to Benefit OVC: Evidence from Kenya and Tanzania, USAID, Measure Evaluation, April 2009 8. Nyangara, F., Obiero, W., Effects of a Community-Focused Approach Supporting the Mosts Vulnerable Children: Evaluation of SAWAKA JaliWatoto Program in Kagera, Tanzania, USAID, Measure Evaluation, August 2009 9. Thurman, T., Hoffman, A., Chatterji, M., Brown, L., A Case Study: Kilifi Orphans and Vulnerable Children Project, Measure Evaluation and Catholic Relief Services with USAID, March 2007 10. Nyangara, F., Thurman, T., Hutchinson, P., Obiero, W., Effects of Programs Supporting OVC: Key Findings, Emerging Issues and Future Directions

  6. Evaluations Surveyed 11. Murombedzi, Cornelius, Mugumya, Firminus, Opportunity International/Habitat for Humanity: Evaluation of Sustainable Income and Housing for Orphans and Vulnerable Children in Africa, International NGO Training and Research Centre, August 2009 12. Larson, B., Hamazakaza, P., Costella, J., Laurence, S., Costing of OVC Service Delivery in South Africa and Zambia, Center for Global Health and Development and Health and Development Africa, USAID/Project SEARCH, September 10, 2010 13. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Final Report, Christian Children's Fund and USAID, March 2008 14. Jaden Consultancies, Weaving the Safety Net: Mid-Term Evaluation, Z Scores Christian Children's Fund and USAID, March 2008 15. Scott, N., Pfaff, M., Semrau, K., Leyenaar, J., Suojanen, A., Mugala, T., Simon, J., End of Project Evaluation Report: Faith-based Regional Initiatives for Orphans and Vulnerable Children (FABRIC), USAID/Project Search, boston University OVC-CARE Project, Family Health International, August 2010 16. Yates, DeeDee, Finding the Inflection Point in Health Education and Economic Strengthening: Final Evaluation of Project HOPE's Track 1 OVC Program Sustianable Strengthening of Families and Orphans and Vulnerable Children Mozambique and Namibia, Cooperative Agreement #: GPO-A-00-05-00019-00, May 31, 2010 17. Capps, J., Bukuluki, P., Davis, S., Swarts, B., Bailey, B., Wakubona, S., Sustainable Community Support for Orphans and Vulnerable Children in Tanzania and Uganda, Final Salvation Army (TSA) Uganda, June 2010. 18. RAPIDS Evaluation Final Report 2005–2009 Key Findings; Population Council, Africare & Rural Net Associates, July 2009, Revised 2010

  7. Overview and data abstraction

  8. Age IssuesFocus on Adolescents 8-14 (6) 0-4; 0-18 (2) 5-11 (1) 7-12; 7-15 (2) 13-18; 13-17 (2) 12-19; 12-17 (2) N/A – (2 costing studies) 3 with unclear age

  9. Grading methodology • STRONG:No study used a RCT design in any form none graded as "strong". • ADEQUATE: Nine studies graded "adequate" -some form of before/after, case control, and post intervention comparison design. • 1 qualitative (with good design) • Two costing studies graded as adequate - utilised consistent audit/costing data. • WEAK: Six studies were graded as "weak" and conclusions/ findings should be viewed with caution.

  10. Quantitative vs qualitative methodology The war is over • 13 used/included quantitative • 11 used/included qualitative • Qualitative is no excuse for poor methodology. • If qualitative used, quality standards on how sampled, how recorded, how analysed – coding and themes – not just the use of quotes (which is what a number did)

  11. Complex interventions • Most programmes are complex – mechanisms of impact difficult to align.

  12. Critical Absence of Baseline Data

  13. Data Handling • Vast number of data points (15 Scott) • Simplistic reporting (failure to do complex analysis) • Percentages versus statistical tests • Power calculations

  14. Theory • Role played (low) • Recourse to theory (eg knowledge does not equate to behaviour – why focus and measure of knowledge eg 9 Nyangara looks at knowledge gaining, some look at attendance at knowledge imparting (not even if they got the knowledge). • Generalisability • Understanding mechanisms

  15. Two forms of Evaluation – Against Pepfar Goals or against Programme goals

  16. Methodological Trends Baseline:PEPFAR wants evidence-based approaches but most evaluations had no baseline, which is the only way to monitor and track impact of dollars and labor invested. Quantitative:Several of the evaluations were able to tell us how many OVC received services, how many caregivers had taken out a loan and repaid it. Standardized Indicators – good methodological reasons for using validated indicators (See Tulane group for good example). Outcome - How do you operationalize efficacy? (eg Yates 16 – use attendance at Health education as the indicator, but what if you attended and slept through? Gender:many mention few analyze

  17. Sampling – good methodological reasons for using excellent sampling and reporting refusal rates. • Staff trainingin each year were a relatively large share of program costs. If such expenses are considered investments in human capital, the training would provide services to the project over more than one year. If there is substantial staff turnover, prior investments in training would not provide services to the project in the future • Dose Issue - Exposure dose seems to be an emerging issue. Home visiting is not a standardised commodity. Need to harmonise concepts in description and quantity standards

  18. Findings by Basic Needs:Food/Nutrition - 14 / 18 9/14 evaluations focused on food/nutrition found OVC households to be moderately or severely food insecure.  Improvements shown, but questions raised about dependence and sustainability.

  19. Health Care ––11/18 8/11 studies demonstrated improved health outcomes Specifically notable when combined with economic strengthening and high dose/follow through

  20. Education–– 15/18 Enrollment effect Supplies positive mental health effect. 0 / 14 identified gender imbalances in school enrollment, although one reported that more girls than boys miss more than two days of school. Cost – resentment Ongoing funding need until government shoulders this.

  21. Psychosocial––13/18 10/ 13 home- visits clearly benefited OVC but not always guardians Negatives associated with infrequent visits, low meeting attendance at Kids' Clubs Targeting carers indirectly benefits OVC Volunteers overwhelmed by caseloads and ill equipped to provide HB PSS counseling, particularly to adolescents and caregivers.

  22. Economic Strengthening–9/18 • PEPFAR's investments in local savings and loan schemes are proving to be effective in raising many OVC households out of poverty • Economic strengthening intersects significantly in other core areas of basic need, including food/nutrition, health, education, psychosocial support and shelter. Thus, economic strengthening remains one of PEPFAR's most potent and effective strategies for improving the lives of OVC and their families • Costing - Great variation for example • Kids Clubs: $1.96 - $14.08 • Home Visiting: $3.55 - $89.68 • Gardening $24.00 - $89.00 • Food $1.39 - $32.36 • Educational Support: $17.23 - $141.32

  23. Child Protection––16/18 Measurable role home-based visits play in reducing child abuse, stigma and social isolation Psychosocial interventions targeting guardians often benefited OVC as well Children whose homes had been visited by a volunteer > birth certificate (36% of all children had birth certificates) Home visitor training associated with improved outcome

  24. Shelter––6/18 Shelter was the least prioritized basic need in all evaluations. Provision of shelter made significant positive differences in multiple areas, including protection, psychosocial support and health

  25. HIV Prevention––15/18 Children's HIV knowledge appeared to increase in 7 out of 13 evaluations. Some interventions appeared to be more thorough than others. School-based HIV prevention activities were part of 5 out of 13 prevention initiatives and were estimated to be cost effective.

  26. Summary Quality Standards. Define PEPFAR’s quality standards for evaluations? Methodology? Standardization? The very mention of "PEPFAR" should be synonymous with quality Costing - Great variation Volunteers -costly cyclical patterns are troublesome to program continuity and quality, heavy reliance, questions of quality. Food3 / 13 focused on food - found > 80% of household samples were severely food insecure (86%; 86.3%; 88.5%). Kids’ Clubs – Mixed findings. Often volunteers poor results on issues of stigma and discrimination. One Stop Shop - Harmonizing and integrating complex PEPFAR-funded OVC programs can yield better and multiple outputs An emergency response should be saved for emergencies Evidence- based programming should be a priority - There was weak evidence of gains and impacts in the five core areas, but with a number of caveats

  27. The way forward • Updated PEPFAR OVC Guidance (2012) stipulates: “Allocate at least 10% of program budgets to M&E” • PEPFAR supporting the development and implementation of: • Methodology to track impact of child welfare and protection systems strengthening • Standardized quantitative program evaluation tool to measure child and caregiver/household outcomes across countries at population level (Will be introduced in Nigeria, Mozambique)

  28. Evidence-Based Policy?

  29. How Good at Doing Good?Reviewing the PEPFAR OVC FactorandPEPFAR HKID Portfolio Review: Children of the HIV/AIDS Epidemic Reports available at: http://resources.ghtechproject.net

More Related