Getting more bang for the buck, euro, baht, dong, yuan, peso, rupee, kip…

1. Getting more bang for the buck, euro, baht, dong, yuan, peso, rupee, kip… The Challenge of Strengthening Impact Evaluation in the East Asia and Pacific Land Portfolio World Bank Land Thematic Group Mini Retreat November 19, 2007

2. Background and context of EAP program • Challenges • Role of impact evaluation • Four examples • Going Forward

3. The Thailand Model has driven most of East Asia’s work in land, at least for land administration • EAP has one of the largest land project portfolios in the Bank, building on the pioneering work in Thailand in the 1970s and 1980s, particularly in land registration • The “Thailand model” has been replicated in Lao, Indonesia, Philippines and Cambodia, and elements are now being extended to Vietnam and China • The Thailand model includes impact evaluation as a core component and has been part of all of the scale ups

4. The projects have similar development objectives in terms of contributing to poverty reduction: Increase land tenure security through more efficient land titling and registration to: • Increase access to credit • Increase incentives to invest in land

5. However, the Thailand approach to impact evaluation has not been scaled up systematically… • While top Bank researchers were involved in Thailand, subsequent projects had very limited involvement of Bank impact evaluation specialists to assist design of this component • While Bank researchers worked closely with local researchers to build their skills in Thailand, there has been little or no methodological support to local research institutes in subsequent projects • Despite having a fairly consistent design with similar development objectives, there has not been an effort to develop standard tools and approaches to ensure at least a minimum standard of impact evaluation and provide some basis for comparability

6. EAP is facing several challenges in scaling up region-wide, as reflected in the portfolio • Previous projects satisfactory and generally sustainable • 6 on-going projects (5 land administration and one land reform) for total of about US$200 million • 3 projects are at-risk generally due to fiduciary issues (legal cov, FM and proct) rather than technical • Significant disbursement lags (12 to 24 months) • 2 new projects (one land reform, one land administration) for delivery in FY08 and 3 for FY09

7. …which has focused us on some common areas for improvement: • Better mechanisms to address systematic corruption at project and sectoral levels • Project designs which are better tailored to country circumstances through simplified design, restructuring and enhanced SPN support • Capitalizing on new opportunities emerging with decentralization and technology innovation • Building effective approaches for improving governance through both supply and demand side approaches for improved service delivery • Defining effective instruments and long term programmatic approaches to the more sensitive aspects of the land agenda, such as land reform and state land issues, to develop stakeholder commitment

8. Monitoring and evaluation are critical to meet these business and development challenges: • Provide more “real time” feedback to identify needs for adjusting programs • Bring increased transparency to support participation and oversight by local communities and focus attention of politicians • Establish more objective, common information base for stakeholder dialogue on tough issues – M&E beyond projects

9. Does impact evaluation matter for EAP’s land work? Yes! • Scale up has taken place in very different country environments than Thailand, suggesting that the benefits themselves and the results chain differ across countries. • In the absence of commonly acknowledged, objective evaluation, dialogue space is often occupied by unsubtantiated critics of the Bank’s support • Performance under different enabling environments may inform the options to be presented in policy dialogue. • There are several aspects to program design which could be adopted more to local context but need better understanding of their role and performance in obtaining objectives. • As Bank role evolves with rapid changes in client countries, we need to understand what is working and what isn’t in developing service delivery mechanisms.

10. What are the key questions to address? • Who are the main beneficiaries of land titling programs? • Are there losers and why? • What is the impact of titling on poverty and growth, and how does it happen? • What are impacts on broader land market development including rentals? • Is full individual title required to obtain the benefits or can intermediate steps provide most of the benefits for the poor at lower cost? • Alternatively, to what extent do limitations on the property right reduce the poverty reduction impacts? • How do efforts to strengthen gender equity in the legal aspects of titling affect the above? • How do we explain differences in poverty outcomes between regions and groups in a country and across countries? • What are the complementary elements of the enabling environment which are required for titling to contribute to poverty reduction and growth?

11. Example 1: Indonesia Land Administration Project –Ex-post without baseline but control group-done • Contracted SMERU, a local research institution • A combination of quantitative survey and in-depth interview was used in the study. A questionnaire was developed by SMERU and used in data collection for the quantitative survey, while in-depth interviews used a guideline. • 14 districts and cities where project operated were chosen, and then subdistricts were chosen based on degree of urbanization. • There were 1,596 randomly selected respondents from the villages and some key informants interviewed in the study. • 1004 were households that received land certificates, 84 were in project villages and were eligible but did not receive certificates, and 508 were outside the project area and were considered the control group.

12. Example 1: Indonesia - Results • The program had no bias in terms of socio-economic groups receiving certificates • Although significant “additional costs” to titling process, majority felt significant net benefits • 70% of certificates in name of husband only for jointly purchased property • 12.8% greater mortgaging • 5.3% higher investment • 1.7% increase in land transactions in areas with certificates • 33% increase in land taxes in areas with certificates

13. Example 2: Philippines Comprehensive Agrarian Reform Program AAA-ex-post using secondary data • Work with Asia Pacific Policy Center – a local research center • Focus on analyzing existing data including (i) 1991 and 2002 agricultural censuses; (ii) the Integrated Farm Household Surveys of 1991, 1997, and 2004 conducted by the Bureau of Agricultural Statistics using the agricultural census as a sampling framework; (iii) the Family Income and Expenditure Surveys (1997 and 2002) and the Annual Poverty Indicators Survey (1998 and 1999) from which a set of panel data on 17,897 households is available; and (iv) a set of panel data collected by the Department of Agrarian Reform as part of its CARP impact assessments in 1990, 2000 and 2007. • Attempt to assess poverty impact, consequences of restrictions on sale and rental of reform lands • Other issues addressed through non-statistical approaches

14. Example 3: China Land Titling Pilot – Ex ante with specific baseline – on-going • Development of questionnaire and evaluation strategy by Scott Rozelle working with Chinese Agricultural University • As one small group in one village is being titled in the pilot project, the following means will be used to choose the sample: a.) select 30 households in the titling small group; b.) select 5 households in each of the other small groups (or a maximum of 20 households); c.) select 10 households in each of two adjacent villages. A total of 30+20+20=70 households will be surveyed. This survey will be repeated each year for the next 5 years. • In anticipation of the upscaling of the project, funding will be sought to do baselines and follow the progress of the upscaling of the project. • It is expected that when the plan is devised for the upscaling, it will maximize the learning about the various ways and options for implementing land titling.

15. Example 4: Vietnam–Land Registration w/Focus on Gender–Ex Ante integration with other surveys • Issues new certificates and changing from man only to joint • this will provide us with a unique possibility to test empirically whether and how joint ownership differs from individual one • Plan to put specific questions into upcoming (May 2008) LSMS on land certificates (in name of male or female), legal knowledge, conflicts, and land market participation • Also Danish TF to have the University of Copenhagen (who are already having a panel survey scheduled for 2008, 10, and 12 to evaluate a DANIDA ag development project) at about the same time to use a survey instrument with identical sections to oversample the intervention provinces and communes

16. Issues emerging from review of impact evaluation in EAP land projects: • Classical results chains but inconsistency in how this is described in terms of expected project impacts and failure to do ex-post assessment of critical assumptions • Lost opportunities to introduce or utilize elements of randomization to strengthen impact evaluation • Problems in definition of control groups – delays with baseline, non-random selection, etc. • Very different survey instruments limit ability to link to household survey results, compare between projects or develop international comparisons • Limited use of other information in country • Absence of cost-benefit or other efficiency measures • Limited focus on other impacts including governance attitudes, land use sustainability and other qualitative data • Need for better measures of institutional devpt impact and sustainability • Limited interest from clients, particularly if no grant funding, delays start up and limits use of results • Impacts exceed project period and bunching of implementation at end • Missed opportunities to build evaluation capacity with local institutions

17. What we are doing to get ahead of this: Strong interest from task teams and support from Land TG, including impact specialists as part of peer review process Complete stock taking and agree on areas for focusing and sequencing strengthening work Simplified project design should lead to more focused evaluation Integration of consistent land modules in household surveys used for poverty assessments Highlighting linkages of land to poverty reduction in AAA More integration of process evaluation to capture governance aspects Broaden instruments to include more PSIA, PETS, citizen report cards and others which can be implemented with local partners Engage govt, NGOs and other stakeholders in evaluation process Develop common instruments and regional community of practice of local institutions around land impact evaluation Seek funding from Spanish TF for technical & implementation support to develop approach in one or two countries which can be replicated

18. Let’s remember that we do this because poor people want tenure security and we want to do it better!