140 likes | 278 Views
Andrey Ivanov , Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC. Monitoring and evaluation of National strategies for Roma Inclusion: What data and for what purpose?. Monitoring and evaluation 101. Basic typology Monitoring (the process)
E N D
Andrey Ivanov, Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC Monitoring and evaluation of National strategies for Roma Inclusion: What data and for what purpose?
Monitoring and evaluation 101 Basic typology • Monitoring (the process) • Evaluation (of the results) • Intermediary or final • Using indicators • Input Output Outcome Impact Applied at different levels • Of the National strategy • Of the Action Plans • Of Individual interventions Monitoring what determines the kind of data and the kind of indicators used
The strategies: How to get results • Having a National Strategy drafted is the beginning, not the end. It needs to be matched by • National Action Plans (usually covering 2 years periods and regularly updated) • Local action plans • Sector specific and integrated projects • In the case of the strategy, for M&E we need • Clear targets– numerical expression of the objectives • Adequate indicators– the definition of the target (how do we measure whether the objective was reached) • Quantitative baseline – the starting point against which the progress/regress is be quantified (the value of an indicator at to) • Milestones – intermediary targets on the way to the general target to keep track of progress (the value of the indicator at t2, t4, t6) • The lower you go, the higher the chances for real inclusion of Roma in the process
Different results at different levels • National strategy • Long-term change in the situation of the target group • Difficult to attribute results (but not impossible) • National action plan • Closer link between inputs and outcomes • Clear objectives (that are the strategy’s milestones • Local action plans • Direct link to project outputs • Clear territorial dimensions • Individual interventions • Counterfactual possible although difficult
Open questions • What targets for individual priority areas? • Roma specific or general? • What baseline? • 2004? 2011? 2013? • What source of data? • Data availability determines the indicators or the other way around? • What milestones? • The link to individual OPs
Another open question: who’s Roma • Politically sensitive (incl. misuse of data for political purposes) • Legal (data protection) or ethical considerations (privacy and fear of stigma) constrains • Insufficient attention to comparability across countries, sub-regions, ethnic groups • The crucial question: what to put in the denominator of an indicator? • The nightmare answer: whatever serves the purpose…
Who is Roma? Possible options • Self-identification • Outside (‘imposed’) identification • By non-Roma • By Roma • Combined (multi-stages) – used in the surveys of UNDP (2004 and 2011) and of FRA (2011) • Crucial decision to be made: are we addressing “all Roma” – or “Roma at risk of marginalization”? The answers is both politically and policy loaded.
Possible source of data for M&E The data set of Roma vulnerable to marginalization generated from the UNDP/WB regional survey that is part of EU Roma Pilot Project funded by DG REGIO and from FRA Roma Pilot Survey: • Monitoring fundamental changes possible (but not short-term fluctuations). Suitable for National Strategy evaluation • Most indicators have a base-line populated by data from the survey conducted in 2004 by UNDP • The “best game in town” (because it’s the only one…) Caveats: • Still a survey (a sample is always a sample) • Expensive, provides data on “Roma vulnerable to marginalization” – and not on “Roma in general” Other options • Roma boosters in HBS • Longitudinal surveys
Going beyond ethnic identity • Be pragmatic - don’t be obsessed by (don’t ask) unanswerable questions like “Who’s Roma?” • But don’t dilute the task of Roma inclusion either • Give priority to socio-economic status • But still keep ethnic identity and specifics in sight • Stick to territorial characteristics • Most of the vulnerable Roma live territorially in separate (segregated) communities • Territorial mapping of those communities is possible • Once a detailed map of Roma-dominated communities is available, it will be possible to correlate ethnic characteristics with territorial tags (individual’s address) • This will allow monitoring a standard set of indicators for a population living in an area with ***% of Roma
The benefits of territorial approach • Makes possible to identify the absolute number of the population and not only a percentage • It can be an option solving the problem of individual respondents refusal to declare ethnicity in the census or to declare different one • Less susceptible to political fluctuations • Is more comprehensive in terms of social inclusion (targeting vulnerability per se) • It grasps the marginalized, visibly excluded segment of the Roma population • Actually reflects the fundamental logic of inclusion (including the excluded, not those included already) • Is best for ensuring that control groups (non-Roma living in the same area) are also included
An optimal compromise • One approach cannot serve all purposes • Apply different data sources for different planning frameworks • National Strategy – EU-wide survey (representative of… - a matter of political compromise) • National Action Plans – territorially-focused mapping • Individual interventions – project outcome evaluation • Integration of the three levels requires clear milestones in strategies and action plans
Conclusions • Integrate the monitoring functions into the entire implementation chain of the strategy • Don’t rely on one source of data and give priority to territorial approaches • Include clear milestones in National Strategies that would serve as a link to the National action plans and OPs • Compete the entire vertical planning and M&E architecture (strategy plan call for proposals interventions) • Go beyond poetry in Operational Programs evaluation building the latter bottom up • Be aware: keeping evaluations vague means keeping them fake