PNPM-Urban Process Evaluation : Key Findings and Considerations

1. PNPM-Urban Process Evaluation :Key Findings and Considerations November 2011

2. Goals for process evaluation

3. Program objectives for PNPM-Urban • Source: Review of World Bank PNPM-Urban III Project Action Document

4. Evaluation focused on six themes emerging from the results framework Assessing the role of CDD • To what extent is CDD realized in planning, decisionmaking and implementation? Are socialization activities sufficiently inclusive? • Is PNPM-Urban meeting the needs of the poor? Links with local government How does PNPM-Urban link with local government and with sectoral programs? Is there much collaboration / duplication? Capacity building and effectiveness of the facilitators • How effective are the current facilitators and do they adequately meet the needs of the communities? What characteristics contribute to strong facilitators versus weak? Quality of infrastructure • Overall how is the quality of the infrastructure investments? What factors contribute to high- and low-quality works? How cost effective are infrastructure investments? Control mechanisms/governance • How effective are the current governance structure/ control mechanisms that are in place? Are they adequate or could they be improved? If so, how? Adequacy of block grants and insights from PAPG/ND • Do communities express a desire for larger grants?. • How do experiences and outcomes vary under PAPG/ND and what are the differential impacts?

5. Analysis draws on multiple sources of information Desk and Literature Review Review of PNPM-Urban MIS Data Subject Matter Expert Interviews Methodology Site visits Formative Research

6. Focus of Data Gathering: Site Visits • Goal was to design and implement efficient data collection effort under time and resource constraints, while capturing • sufficient number and variation in sites, including 3 ND • subjective perceptions of multiple, different stakeholders • objective assessments of PNPM-Urban project output quality • Final fieldwork design reflects important trade-offs • 16 field sites to be covered in 5-6 week timeline • priority focus on infrastructure • simultaneous dispatch of four field teams led by RAND and SurveyMeter, with four engineering teams after heavy pre-field preparation + common initial training and 2 site visits • Fieldwork commenced May 28 through June 25, 2011.

7. Site Selection • Randomly selected from IFLS communities (80% of national population) with projects in the MIS database > 2 years old • Regional stratification by urban population/PNPM-Urban funds, wealth stratification by IFLS community wealth index

8. 360-degree approach to data collection PNPM-Urban Management • District Program manager • Korkot / Askot Government • Bappeda and district MPW leadership • Kota Officials Local PNPM-Urban • Facilitators • Consultants/planners Local Government (and other civil society) • Camat, Lurah, RT/RW leaders • NGO representatives PNPM-Urban community leaders • BKM leader • Infrastructure KSM members Community members • Poor households in general • Infrastructure project beneficiaries

9. 360-degree approach to data collection PNPM-Urban management and government • Key informant interviews Local PNPM-Urban, government and other civil society • Key informant interviews PNPM-Urban community leaders • Key informant interviews • Rapid survey about BKM and KSM • Rapid survey of poor households • Male and female focus groups + interviews • General poverty • PNPM-Urban • Infrastructure beneficiary interviews Community members Other • Infrastructure assessments • MIS data assessment

10. Key Findings 1: Community socialization and participation • What is working well? • Knowledge of processes among local leaders and individuals within PNPM is good • Community institutions like the BKM largely function as intended • High level of recognition of PNPM-Urban projects among community (although not always PNPM-Urban) • High level of expressed satisfaction with project outputs among community and beneficiaries • High levels of community contribution to implementation, although role of paid and unpaid labor appears to vary

11. Key Findings 1: Community socialization and participation • What are areas for improvement or consideration ? • Little process awareness among community members: rarely critical but tend to approve /ratify rather than propose project. Decision-making tends to proceed by delegation. • Stated preferences = demand for social and economic projects: respondents identified loans/training as most important needs • Targeting the poor via infrastructure projects challenging, because of geographical spread • Voluntary administrative burden is taxing for BKM and KSM, may affect selection of leaders • Women’s participation in decisionmaking is relatively low, potentially due to exclusionary norms and protocols

12. Key Findings 2: Links with Local Government • What is working well? • Involvement by local government varies across sites, from little substantive involvement (e.g., only signing off on projects) to substantial local government support (e.g., participating in decision making and oversight) • Even in low-interaction sites, Musrenbang process is used effectively to convey information about efforts: there appears to be little or no duplication of activities or priorities. • What are areas for improvement or consideration? • Few respondents mentioned training for local governments • Observe coordination, but no widespread collaboration or joint planning; some evidence of competitive feeling • Support for more integration is not universal.

13. Key Findings 3: Facilitators • What is working well? • Consensus view across respondents is that facilitators are playing a significant part in all stages of PNPM-Urban, and that facilitators’ contributions improve the quality of projects • Rotation of facilitators limits cronyism and increases the exposure of skilled individuals • What are areas for improvement or consideration? • Facilitators universally appear overworked and overextended. The existing administrative burden on the facilitators for compliance purposes is substantial • Most cannot contribute much to encouraging participation at the sub-Kelurahan level. • Timing of rotations is disruptive to the project cycle

14. % Infrastructure Spending

15. Key Findings 4: Infrastructure Assessment • What is working well? • Assessments of PNPM-Urban project quality were high • Subjective assessments compare PNPM-Urban project quality favorably to government projects; respondents attribute higher quality to participatory approach and avoidance of government management  (including some government officials) • Community reports being more likely to maintain PNPM-Urban projects (although too early to objectively determine) • Average levels of quality inspection reported in MIS tally with average independent site assessments, suggesting standards are adequate by local professional standards

16. Key Findings 4: Infrastructure Assessment • What are areas for improvement or consideration? • Most beneficiaries were not consulted or given opportunities to provide feedback or correction • Although generally quality is perceived to be high, some respondents suggest voluntary contributions may lead to the use of lower quality materials • Use of paid versus volunteer labor varies across sites, with some explanations e.g responding to the technical demands of the project, but reflects potential trade-off • Actual projects can depart from plan and MIS reports (also related to control) without the appearance of reliable systematic protocols for reporting(e.g. cases of splitting projects, or bundling types of expenses together)

17. Key Findings 5: Control Mechanisms and Governance – Funds Management • What is working well? • Very few reports of complaints related to misuse of funds • Auditing is generally completed on schedule and uncovers instances of inadequate project performance, selection and implementation at an early stage by facilitators. • Most sites reported relying on the combination of multiple layers of audit, social norms and community awareness of projects to ensure accountability • What are areas for improvement or consideration? • Difficult to assess extent of reporting bias • Previously noted: heavy burden of compliance, issues of control (versus governance)

18. Key Findings 5: Control Mechanisms and Governance – MIS • What is working well? • Relative to its initial goals, MIS performs well in making public a vast quantity of timely and usable data for tracking purposes • Ability to analyze MIS has increased significantly thanks to motivated and highly-familiarized PNPM-Urban technical staff • What are areas for improvement or consideration? • Significant inherent challenges including harmonization of file formats across funders and cycles, training and motivation of data entry personnel at the community level • Substantial room to realize full potential of MIS data for process evaluation: data quality and validation issues remain, many key outcomes e.g. poverty, population are only self-reported, users need significant guidance

19. Key Findings 6: Block Grant Size and ND Model • What is working well? • Relatively few sites reported size was inadequate for needs (however, this may be biased by local expectations) • Large % of ND site funding is spent on planning and marketing, suggesting recognition of problems of participation • What are areas for improvement or consideration? • Larger grants may be used to undertake more small projects in different areas, not realizing economies of scale/scope. (One ND site even desired more flexibility to do so) Concerns expressed that larger projects are harder to manage and difficult to distribute benefits equitably. • ND site planning and marketing efforts not viewed as useful (although sample is small)

20. Limitations of this study (planned and unplanned) • Reiterate that focus is largely on infrastructure • Short timeline precluded more intensive focus group discussions and community member interviews on specific issues that arose during evaluation e.g. expectations of community regarding delegated versus direct participation • Resource and time constraints prevented larger sample of sites, including PAPG sites, potentially more meaningful sub-group analysis • Cost-effectiveness analysis and analysis of sectoral program links could not be completed within the scope of the project • MIS data provided useful insights into program at a high level but precluded robust regression and subgroup analysis; most analysis draws on subjective perceptions, should be interpreted as such.

21. Implications for Design and Experimentation Assessing the role of CDD • Implement universal invitations to meetings particularly for women. Consider alternative methods of eliciting preferences of poor households Links with local government Continue to focus on training and capacity building, emphasizing collaboration rather than competition and potentially consider more structured interactions Capacity building and effectiveness of the facilitators • Build more facilitator capacity through recruitment and/or training that includes interpersonal skills building, reconsider rotation schedules Quality of infrastructure • Allow more direct channels of feedback for those likely to be affected. Improve balance between local responsiveness and control. Consider tradeoffs between community and hired labor. Control mechanisms/governance • Maintain multiple checks and balances with audit system incorporating local participation; increase attention to potential of MIS from bottom data entry to high-level database structure and user interafce Adequacy of block grants and insights from PAPG/ND • Consider individual needs of communities and continue to monitor development of ND projects as they mature