Monitoring and Evaluation: quantity and quality indicators

1. Monitoring and Evaluation: quantity and quality indicators Susanne Milcher Specialist, Poverty and Economic Development UNDP Regional Centre Bratislava (17 September 2004)

2. Outline • Experts’ Group Meeting • Indicators for M&E • The baseline survey • Conclusion

3. Experts’ Group(including one member from the Statistical office, one member from the respective “Minority issues” councils; participation of Roma envisaged) Objective: Within 2-3 years achieve disaggregated data collection capacities at the country level Major tasks: • Elaborate methodologies for sustainable data collection on vulnerable groups • Coordinate the base-line surveys • Elaborate possible approaches to overcome legal barriers • Prepare guidelines for training of Roma activists to participate in data collection and analysis

4. Recommendations by Data Experts’ meeting I The Census can be used to collect ethnic data but • Method improvement: partnership with local communities, instructions • Identification: Multiple choice question on ethnicity; adding question on religion, language, partner’s ethnicity or country of birth; origin HBS/LFS can only partially be used to collect ethnic data • Better than administrative registries • Sampling difficult

5. Recommendations by Data Experts’ meeting II Indicators should be: • Simple: They are easy to use, simple to understand and easy to operationalise (technically feasible). • Valid: They are significant, relevant, and directly linked to programme objectives. • Reliable: They are robust, accurate, replicable, verifiable and objective. • Sensitive: They are capable of demonstrating and capturing change in the outcome of interest (responsive to changes). • Direct: They should measure the result they are intended to measure as closely as possible. Proxy indicators are used as alternative measures if a direct measure is not possible. • Quantitative: They are based on precise and replicable measurements and comparable. Qualitative indicators should supplement quantitative indicators with richer information. • Disaggregated: They are disaggregated by sex, age, education, location or other dimensions and give a better picture about impacts of targeted interventions.

6. Indicators for monitoring and evaluation • Quantitative & Qualitative • Outcome versus input indicators • Monitor impact of policy = aggregate development change • Impact: Have policies resulted in lasting improvements for target population? • Effectiveness: Are the desired results or objectives being achieved? • Relevance: Does the policy meet beneficiaries’ needs?

7. UNDP approach to the issue Reliable quality quantitativedata is a necessary precondition for relevant policies. It means data, which is: • Relevant, adequately reflecting reality, involvement of Roma communities • Comparable – both between countries and with majority populations (control group) in individual countries – over time • Respecting privacy – making sure will not be misused, individual is protected against discrimination

8. The survey I • Supposed to provide base-line data for the “Decade” progress monitoring and for national policy purposes • Covers all countries in SEE and CEE with sizeable Roma minorities (“Decade +”) • Where relevant, has IDPs and refugees boosters • Will be the basis of a “regional vulnerability report” • Could be used as a pilot for similar data collection exercises in the region

9. The survey II • The unit of analysis – household • Main interviewee – head of the household • Universe studied – households in Roma settlements • Roma settlements – municipalities or neighborhoods with high concentration of Roma • Territorial unit – municipalities with share of Roma population equal or above the NTL average as registered by the census

10. The sampling model assumptions • Census understate absolute numbers but reflect the structure and distribution (“where those people are?”) • The major disparities visible at the level of municipalities • Comparability with the “majority in proximity” more important than with national average • Majority boosters – a “benchmark” sample for comparisons with non-Roma in similar socioeconomic environment

11. Quantitative indicators • MDG indicators: poverty rate, enrollment rate, maternal and infant mortality rate, access to water and sanitation • Social exclusion: (long-term) unemployment rate, ethnic and gender ratio of unemployment, items in household • Access to health and credit services, land • Additional: Educational attainment, completion, functional literacy, sources and levels of income, employment per sector disaggregated by age, sex, income status of the household and sub-national level

12. Conclusion • Data and data collection provides the basis for sustainable monitoring and evaluation • Inclusion of “impact and outcome indicators” into national strategies from the beginning • Targets and benchmarks • Control group

13. Thank you! Bratislava Regional Center 35 Grosslingova 81109 Bratislava, Slovak Republic +421 2 59337 111 www.undp.sk http://roma.undp.sk http://mdgr.undp.sk