Assessing Information Needs and Survey Alternatives

1. Assessing Information Needs and Survey Alternatives Kathleen Beegle, DECRG Poverty and Inequality Course Module 1: Multi-topic Household Surveys January 23, 2008

2. The Demand for Data • Performance-based management has created pressures on developing countries to improve the quantity and quality of their macro and micro-data: • Is the public sector delivering good services? • Are country policies/poverty reduction strategies reducing poverty? • Is aid supporting poverty reduction? • In the World Bank: “Results-based” CASs, for example

3. The Demand for Data • Performance-based management • Millennium Development Goals (MDGs) reflects the strong commitment on the part of the international development community and national governments to monitor and evaluate the results of their policies and programs (United Nations, 2000).

4. The Demand for Data • Performance-based management • MDGs • HIPC, Poverty Reduction Strategies (PRSPs) The HPIC (debt reduction initiative for heavily-indebted, low-income countries) requires • a plan for poverty reduction • that can be measured and monitored • access to concessional lending from the World Bank and International Monetary Fund (IMF/IDA, 1999a and 1999b).

5. The Demand for Data • Much of the increased demand for data has been focused on household-level data. • It is not possible to monitor PRSPs or MDGS without solid household data • Measurement of welfare and other key social indicators. (Muñoz and Scott, 2005). • Decisions on appropriate policies to reach the MDGs or the targets in the PRSPs need household data. • Although they do not exclusively need/require only household-level data, although they often imply survey data

6. What these demands look like • PRSP • Measure welfare/poverty • Identify problems--magnitude, causes • Alternative policies • Cost/benefit • Monitor • Evaluate

7. What these demands look like • PRSP • MDGs MDG 1: Eradicate extreme poverty and hunger MDG 2: Achieve universal primary education MDG 3: Promote gender awareness, empower women MDG 4: Reduce child mortality MDG 5: Improve maternal health MDG 6: Combat HIV/AIDS, malaria and others MDG 7: Ensure environmental sustainability MDG 8: Develop a global partnership for development

8. What these demands look like • PRSP • MDGs • General Demand • Poverty and Inequality • Benefit Incidence Analysis • Public services • Determinants of observed outcomes • Assessment of alternative policies • Impact Evaluation • Inputs to Program Design

9. Recent Efforts to Increase Data: Quantity and Quality • Partnership in Statistics for Development in the 21st Century (PARIS21) was initiated to “…to act as a catalyst for promoting a culture of evidence-based policymaking and monitoring in all countries, and especially in developing countries.” (PARIS21 web site). 1999 • Trust Fund for Statistical Capacity Building (TFSCB), managed by the World Bank, set up in 2000 to help build the capacity of statistical systems/ statistical plans

10. Recent Efforts to Increase Data: Quantity and Quality • World Bank line of credit for Statistical Capacity Building to support countries in the implementation of these statistical master plans. • The Program to Improve Surveys of Living Conditions in Latin America (MECOVI), (co-sponsored by the IADB, ECLAC, WB, regional program to improve quality and data • SPARC- Eastern Carribean Initative to improve surveys, UNDP, CDB, WB IADB, OECS, inter alia. (2004) • LSMS Phase IV: Methodological Research on measurement, field work and technological advances

11. Focus of Presentation • Assessing Information Needs • Sources of data • Need for multiple sources of data • Role of household surveys

12. Assessing Information Needs

13. Data Sources • National accounts • Current public expenditure statistics • Program of Price collection (cons./prod.) • Administrative Records (from line ministries) • Qualitative Work • Surveys: • Multi-topic w/ welfare focus (LSMS/IS) • Monitoring (CWIQ, PS) • Income and Expenditure (IES, HBS) • Single topic (Labor Force Surveys (LFS) • Demographic and Health (DHS)) • Enterprise • Facilities

14. Surveys and Policy Analysis Gov’t Programs Social Goals Conditional Cash Transfers Day care centers Public Health Campaign Increase enrollment Increase female LFP Lower infant mortality Households Individuals Firms

15. Surveys: Going Beyond RatesUnderstanding secondary school enrollments, 12-18 year olds, Albania 2002 • In almost all countries we have a single statistic: mean enrollment at the national level. In this case it is 61%. • This is interesting for monitoring purposes, but it doesn’t say much about poverty or other factors. • ... A regional disaggregation would be useful Average Percent

16. Understanding secondary school enrollments, 12-18 year olds, Albania 2002 • In some countries we have regional breakdowns, with marked contrasts • The contrast between urban and rural rates emphasizes the disadvantages faced by rural communities. • Other breakdowns would be useful Urban Average Percent Rural

17. Understanding secondary school enrollments, 12-18 year olds, Albania 2002 • …possibly, official statistics can add the gender dimension • …the figures show that, in urban areas, there is no gender differential but a large gap in rural areas. • But we still don’t know much about who sends their children to school Male Urban Female Average Percent Male Rural Female

18. Understanding secondary school enrollments, 12-18 year olds, Albania 2002 Female, urban Male, urban Male, rural Female, rural Average Percent • …With a survey we can show enrollment rates broken down by consumption level--and thus understand an additional dimension Consumption quintile

19. Gathering information through surveys (household or other) • There is a range of options • They can be ordered along two main dimensions: • degree of representativeness • subjective/objective dimension

20. Degree of Representativeness Case study Purposive selection Quota sampling Small prob. sample Large prob. sample Census

21. Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Structured interview Case study Purposive selection Quota sampling Small prob. sample Large prob. sample Census Open meetings Conversations Subjective assessments Subjective/Objective Dimension

22. Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Structured interview Case study Purposive selection Quota sampling Small prob. sample Large prob. sample Census Open meetings Conversations Subjective assessments Wonderful World of Surveys Census Household Budget Survey LSMS/ IS CWIQ/PS Participatory Poverty Assessments Sentinel Site Surveillance Participant observation Beneficiary Assessment Community Surveys Windscreen Survey

23. Direct measurement Questionnaire (quantitative) Questionnaire (Qualitative) Structured interview Case study Purposive selection Quota sampling Small prob. sample Large prob. sample Census Open meetings Conversations Subjective assessments Wonderful World of Surveys: “Statistical Surveys” Census Household Budget Survey LSMS/ IS CWIQ/PS

24. Features of a Statistical Survey • Structured Questionnaire • Random/Probability Sample

25. Tradeoffs to Consider When Planning a Survey as Part of a System of Surveys • Overall scope • Single vs. Multi-topic • Probability vs. Purposive Sampling • Sampling vs. Non-Sampling Errors • Time vs. Cost • Data vs. Capacity Building • Surveys over time: repeated cross sections, panels, rotating

26. Summary • Surveys are one source of information among many (system of information) • Consider all the key elements of a National Poverty Monitoring System

27. Key Elements of a National Poverty Monitoring System • Timely Annual National Accounts • Current Public Expenditure Statistics • Program of Consumer and Producer Price Statistics • In-depth Welfare Survey (LSMS/IS, IES?) • ‘Light’ Annual Monitoring Surveys (CWIQ, PS) • Longitudinal Studies • Qualitative work on key topics • Specific tools for project/program/policy monitoring and evaluation

28. Summary • Surveys are one source of information among many (system of information) • No one survey can meet all data needs: System of Household Surveys

29. System of Household Surveys • Goal: System able to respond to evolving needs: not produce data X or survey Y • Determine data needs before they are URGENT • Identify appropriate instruments, • Implement them properly, timely fashion, • Analyze the resulting data

30. Improving the SHS • Linking Users and Producers • Providing adequate resources • Continuous Survey Program • Not necessarily permanent survey • Benefits • Avoid loss of capacity • Create greater levels of capacity (building on existing) • Economies of scale • Policy makers know when data will be available • Protects NSO from pressures for ad hoc surveys • Ongoing system actually allows more flexibility and responsiveness

31. Summary • Expanding demand for timely, relevant data • Need to determine the range of data needs to begin to define a system of information • Surveys are one, important, source of information among many • No one survey can meet all data needs: System of Household Surveys

32. References • United Nations (2000). “United Nations Millennium Declaration.” United Nations’ General Assembly, Fifty-fifth Session, New York, New York. • International Monetary Fund and International Development Association (1999a). “Building Poverty Reduction Strategies in Developing Countries.” Report to the Board of Directors, International Monetary Fund and International Development Association, Washington, D.C. • International Monetary Fund and International Development Association (1999b). “Heavily Indebted Poor Countries (HIPC) Initiative: Strengthening the Link between Debt Relief and Poverty Reduction.” International Monetary Fund and International Development Association, Washington, D.C. • Muñoz, Juan and Kinnon Scott (2005). “Household Surveys and the Millennium Development Goals.” Paris21, processed.

33. References • DECRG, (2006) “LSMS IV: Research for Improving Survey Data”, processed • LSMS Web page: http://www.worldbank.org/lsms/ • ISLC/ MECOVI web site: http://worldbank.org/lac then search on MECOVI