Namibia looking f orward: Effectiveness of capacity development support to local partners

1. Namibia looking forward:Effectiveness of capacity developmentsupport to local partners Stakeholder data validation workshop July 2, 2012Windhoek

2. Why we are here today • To share Pact Namibia’s internal evaluation findings with stakeholders • To get stakeholder validation of the evaluation recommendations

3. Agenda

4. Background • USAID Namibia began scaling up HIV programs in 2003. • In 2007 USAID awarded Pact a 5-year project which included grants, organizational and technical assistance to CSOs, predominantly in areas of: • HIV/AIDS prevention • Care and support to orphans, HIV-vulnerable children and home-based clients • Strengthening capacity of the Ministry of Gender Equality and Child Welfare

5. Background • In 2010 – award changed to focus on two objectives: • Strengthening capacity of the Ministry of Gender Equality and Child Welfare to effectively coordinate activities relating to gender equality and orphans and vulnerable children • Strengthening six civil-society organizations to be eligible for U.S. direct funding while maintaining high standards of existing services

6. Tools to strengthen civil society • Organizational Development (OD) Roadmaps measures CSO partner organizational systems and structures objectively • Basis for identifying organizational efficiency gaps and prioritizing interventions • Progression on a scale from 1 (nascent) to 5 (mature) • Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity-building activities

7. Tools to strengthen civil society • Comprehensive Institutional Strengthening Plans to identify, schedule and monitor all capacity-building activities • Organizational Development (OD) Roadmaps • 10 capacity areas: Purpose & planning HR management Monitoring & evaluation Networking Governance Organizational sustainability Financial management Grants & compliance Operations management Projects & services

8. Organizational Performance Index (OPI) Scored on a scale of 1 to 4on each of the following dimensions: • Effectiveness.Achieving outcomes, meeting standards • Efficiency.Delivering services, increasing reach • Relevance.Engaging target populations, promoting learning • Sustainability.Mobilizing resources, increasing legitimacy

9. Why do this evaluation? • Assess the sustainability of graduated partners’ capacity development • Review the quality and quantity of services offered to beneficiaries of former and current CSO partners (partially done) • Assess the effectiveness of the Pact Namibia capacity building assistance to the partners that have graduated or plan to graduate • Draw lessons for other Pact programs that are facing similar challenges of graduating partners

10. Evaluation questions • To what extent was Pact Namibia’s capacity development support effective? • To what extent were partners ready for and able to effectively manage direct funding? • To what extent did service quality and coverage change over the last two years?

11. Process 3 phases: • Designed terms of reference and data collection tools (virtual group) • Collected data (Pact staff from 5 country offices, for neutrality) • Analyzed data, drafted and disseminated report (current group of Pact staff)

12. Data collection tools • Partner survey (Mobenzi) • 17 current & former partner organizations interviewed • Quantitative & qualitative • Organizational Performance Index (OPI) • 12 of 17 partners • Pact staff survey (Mobenzi) • 4 core Pact Staff from each technical area • USAID tool • Historical OD Roadmap scores

13. Data analysis • Quantitative data: • Merged datasets and did selected frequencies with statistics and cross-tabulations using SPSS and Excel. • Qualitative data: thematic analysis.

14. Partner breakdown

15. Study limitations Sample size • Only 17 organizations • Limited number of respondents in some organizations; some interviewed were previous staff Time • Could not conduct survey of beneficiaries

16. Presentation of the findings • Capacity development impact • Graduation to direct USAID Funding • Lessons learned for capacity development organizations • Preliminary recommendations

17. Capacity development impact

18. Key areas of analysis • How did length of support influence partners’ OPI scores and perceptions of usefulness of support? • How do Pact and partner perceptions of support compare? • How do OD Roadmap supported partners differ from other partners? • Which areas of Pact support did partners rate as most useful and of highest quality? • What were the most significant changes resulting from Pact’s support?

19. 1. Partners scored longer support less useful Monitoring & evaluation Strategic Financial Program thematic

20. 2. But Pact scored longer support more useful M&E Strategic M&E Strategic Program thematic Financial Program thematic Financial Partners score longer support lessuseful Pact scores longer support moreuseful (except financial)

21. 3. Longer support – higher OPI scores Program thematic Monitoring & evaluation Financial Strategic

22. 4. OD Roadmap partners scored higher on OPI

23. 1a. But non-graduates more keen on financial management Non-graduated partners

24. 5. Non-OD Roadmap partners rated Pact support more useful Non-OD Roadmap OD Roadmap

25. 6. Partners rated M&E support highest Usefulness rating Quality rating

26. 7.Most significant changes M&E and Financial management the “cornerstones of programs” (partner, Pact and USAID responses) • M&E • Timely reporting • Capturing the right and required data • Use data for decision making • Finance • Proper management of organization finance & USAID funds • Complete and accurate finance reports • Proper budget management

27. 7. Most significant changes • Moved from no system to very good system in all areas (partner responses) • Buy-in from organizations (Pact responses) • How do we sustain the change? • Experience handover and staff retention • Resources

28. Summing up • More years of support = lower usefulness rating, but higher OPI scores • Pact and partners see usefulness differently over the duration of support • OD Roadmap support = lower usefulness rating, but higher OPI scores • M&E support rated most useful and of highest quality • Most significant change in M&E and financial management

29. Graduation to direct USAID funding

30. Key areas of analysis • Capacity-building support that partners would like more of before graduation • Graduate vs. non-graduate performance on OD Roadmap and OPI • Prerequisites for graduation • Key capacity-building support for graduated partners

31. 1. Partners want more technical support before graduation

32. 1b. And graduates far more interested in technical support Graduated partners Non-graduated partners

33. 2a. Graduates scored higher on OD Roadmap Graduates Non-graduates MERL Purpose & planning Financial HRM Programs Governance Operations Compliance Sustainability Networking

34. 2b. But OPI not different Graduates Non-graduates Efficiency Relevance Effectiveness Sustainability

35. 3. Graduation prerequisites from Pact and partners • Strategic plan • Governance • Strong financial systems and internal controls • Administrative systems, including human resource management • Skilled staff and technical capacity • Strong monitoring and evaluation

36. 4. Key post-graduation capacity development Priority • Resource mobilization support • Technical/Program support As needed • Monitoring and evaluation • Database development • Evaluations • Routine data quality assessments • Financial management

37. Summing up • Graduated partners would have liked more technical support before graduation • Graduated partners averaged 3.4 on Roadmap, up from 2.5 for non-graduated partners (no difference on OPI) • Six graduation prerequisites: strategic plan, governance, finance, admin, technical capacity, M&E • Key post-graduation needs: technical and resource mobilization support

38. Lessons learned for capacity development organizations

39. Key areas of analysis • Retention of systems after capacity building,and impact of who decided to engage in it • OPI scores and trends across partner subgroups • Different capacity needs identified by partners and donors • Challenges and advantages of graduating to direct funding

40. 1a. OD Roadmap-supported partners more likely to retain systems OD Roadmap Support Non-OD Roadmap Support

41. 1b. Retention likelier if system is partner-driven

42. 1c. Retention likelier if system is partner-driven Strategic Monitoring & evaluation Financial Program thematic

43. 2. OPI scores • Average OPI scores show lower resource mobilization but high social capital • Graduated partners generally score higher • Current partners score low on resource mobilization and service delivery

44. 2a. Average OPI shows lower resource mobilization, high social capital Effectiveness Efficiency Relevance Sustainability Average OPI

45. 2b. Graduates need to focus on results and external evaluation of standards OD Roadmap graduates Effectiveness Efficiency Relevance Sustainability Average OPI

46. 2b. Current partners scored lower on resource mobilization and service delivery Non-graduated OD Roadmap partners Effectiveness Efficiency Relevance Sustainability Average OPI

47. 3a. Different stakeholders, different top-ranked capacity needs

48. 3b. Graduated partners & USAID: contrasting expectations

49. 4. Graduates found both challenges & benefits

50. Summing up • Current partners most likely to retain M&E, financial and technical systems • Sustainability of capacity building depends on whose decision it was to engage in capacity development • On average partners tend to score lower in resource mobilization on OPI scoring • Partners still need to develop capacity, but are struggling to understand how to access support under direct funding.