SELECTED SLIDES FOR M&E GROUP

1. SELECTED SLIDES FOR M&E GROUP

2. Evaluation Planning Implementation Evaluation is part of the Management Cycle 2

3. Complementary Functions • Without monitoring, evaluation cannot be done well. • Monitoring is necessary, but not sufficient for evaluation. • Monitoring facilitates evaluation; evaluation generates additional data collection and information. • An evaluation of an intervention will often lead to improvement of plans. • An evaluability exercise often results in modifying data collection for monitoring purposes. 3

4. Evaluation is answering • Whether we are Doing the Right Things • Rationale/justification • Client satisfaction • Whether we are Doing it Right • Effectiveness/coherence • Efficiency: optimizing resources • Whether there are Better Ways of Doing it • Alternatives • Good practices • Lessons learned 4

5. Monitoring • Definition • A process of tracking or measuring what is happening. • It includes: • Performance monitoring - measuring progress in relation to an implementation plan; • Situation monitoring - measuring change in a condition or a set of conditions or lack thereof (e.g., changes in the situation of women or changes in the broader socio-economic country context). • Purpose • Measure progress according to plan. • Trigger timely adaptation of response, particularly in crises or unstable contexts; • Generate critical information for the evaluation; • Provide accountability for implementation; 5

6. Evaluation • Purpose • Supports decision-making • Improve programme performance; • Learning by doing; • Provide accountability for programme results. • Definition • Assess the contribution and worth of an intervention. • Judgment is based on criteria such as relevance, efficiency, effectiveness, impact and sustainability. • Evaluation may be conducted during an intervention (at end of phase or mid-point) and/or at the end of the intervention (ex post). • Generate lessons learned. 6

7. A result is a measurable or describable change arising from a cause and effect relationship. What is a result? Sustainable 7

8. Two major elements: 1. Change • Measurable/ describable transformation • Of rights holders’ status • Of beliefs, behaviours and practices • In institutional performance • In quality & availability of Services and other development tools (products) – knowledge, skills A result is NOT the completion of activities!! 8

9. 2. Causality Cause and effect relationship between an action and the results achieved. “If-then” logic. then Strategic Result if then Programme Result if Result then Result if 9

10. Hierarchy of Results Development bottlenecks Desired changes in: • Rights holders’ status - Impact • Beliefs, behaviour and practices – Outcome • Institutional performance – Outcome • Quality & availability of Services and other development tools (products) (knowledge, skills) – resulting from completion of activities of development interventions - Output 10

11. Types of Change Impact Changes in the lives of people: realisation of their rights Institutional Change: values, laws – associated with institutional performance, new institutions Behavioural change: new attitudes, practices Outcomes Operational Change: products and services, knowledge, skills resulting from completion of activities of development interventions. Outputs 11

12. Definitions • Increased Primary education completion; • Pupils with enhanced communication & life skills • BUT overloaded secondary education system Positive and negative long-term effects on identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended. . Impact e.g. Long lasting Changes in the situation (life) of Claim Holders (Children, women, other vulnerable groups) or of their environment The intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. Outcomes represent changes in development conditions which occur between the completion of outputs and the achievement of impact. Institutional Change: values, ethic, rules, laws, practices, performance Or Cognitive & Behaviour change: resulting from knowledge, skills acquisition, practices (individual, society) • Increased retention (particularly of girls) • Increased enrolment (girls and boys) • Improved quality of teaching/learning Outcome Operational Change: products and services, Knowledge, skills resulting from completion of activities of development interventions Two Primary schools in each of the focus districts meet Child-Friendly School standard Products & services resulting from completion of activities of development interventions. Output 12

13. Definitions • More children likely to be living in a protective, nurturing environment; Positive and negative long-term effects on identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended. . Impact e.g. Long lasting Changes in the situation (life) of Claim Holders (Children, women, other vulnerable groups) or of their environment The intended or achieved short-term and medium-term effects of an intervention’s outputs, usually requiring the collective effort of partners. Outcomes represent changes in development conditions which occur between the completion of outputs and the achievement of impact. Institutional Change: values, ethic, rules, laws, practices, performance Or Cognitive & Behaviour change: resulting from knowledge, skills acquisition, practices (individual, society) • New protection policy adopted by government • Implementation institutions at national and • sub-national levels created/ strengthened • National protective environment increasingly in • line with recommendations of CRC Outcome Operational Change: products and services, knowledge, skills resulting from completion of activities of development interventions • Child protection policy drafted • Key decision makers convinced and committed Products & services resulting from completion of activities of development interventions. Output 13

14. Outputs are deliverables Operational changes: new skills or abilities, the availability of new products and services Must be achieved within the programme period Managers have a high degree of control  If the result is mostly beyond the control or influence of the programme or project, it cannot be an output Failure to deliver is failure of the programme or project 4 to 6 outputs per CP outcome Unless under a joint programme, outputs are NOT collective results Outputs: definition & key features 14

15. Test for “Outputs”: Is it mostly (>80%) in your control? Will you take singular accountability for it? Will your funds be used to achieve this result fully? If achieved, will you take the credit for it? If not achieved, will you take the blame for it? If the answer is yes to all five questions above, then it is an output. If not, think again. 15

16. The 2nd LEVEL RESULTS of UN system cooperation Represent changes in institutional performance or behaviours A strategic contribution to the achievement of UNDAF outcomes 6 to 8 only per UNDAF outcome Accountability may be single agency, but is increasingly shared Produced by the combined effects of the CP outputs CP Outcomes:Definition & key features 16

17. Assumption Assumption Assumption Often quite unpredictable/ even more risk Probable/ less risk Less Probable/ more risk Inputs Activities Outputs Outcomes Impact Change in realisation of rights Implementation partners Plugging capacity gaps of DB/ CH instts Improved performance of DB and/ or behavioural change in DB/CH Decreasing Management Control.. Increasing external influences Increasing difficulty in demonstrating attribution Attribution Contribution 17

18. An UNDAF programme result at the outcome level. Achievement depends on the contribution of more than one partner A strategic UNDAF result at the impact level. Achievement depends on joint UNDAF outcomes plus inputs from other partners Joint or individual agency result at the outcome level Result Result Result Result Joint or individual agency results at the output level Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result Result 18

19. Not delivered Achievement of UNDAF results depends on the successful delivery of all partners’ contributions Partnership agreements needed? Advocacy needed?  Result X  Result Result Result  X X   Result Result Result Result Result X X Result Result Result Result Result Result Result X X Result Result Result Result Result Result Result Result Result X X 19

20. Existence of reliable HIV surveillance system Universal access to condoms HIV surveillance system designed (WHO) Policy on condoms agreed among major stakeholders (UNFPA) Regular and adequate supply of condoms guaranteed (UNFPA) Voluntary and confidential testing and advice available Enabling environment exists that promotes risk free behaviour All large employers have policy and campaigns on HIV in workplace (ILO) Policy on VCT agreed among major stakeholders (WHO) National and sub-national bodies exist that oversee implementation of AIDS prevention activities. (UNDP) Health workers able to conduct VCT (UNICEF) A hypothetical UNDAF Results Framework UNDAF outcome 25% reduction in new HIV infections by 2018 90% of people practicing safe sexual behaviour High-risk population groups know how to prevent HIV infection (Europ.Union) 90% of young people 12-18 know how to protect themselves (UNICEF) CP-level outcomes Multi-media campaign messages reach 90% of all people (UNESCO) CP-level outputs 20

21. Action Language servicesprovider’s Perspectives: Unclear success Criteria Emphasis on Processes (activities) Change Language Rights holders/ Stakeholders’perspective: They can internalize and own it success Criteria: clearer Emphasis on change (of capacities and status) Key Principle # 1: LanguageAction Vs Results Language 21

22. Refining Results • Action Language • To strengthen the capacity of teachers to teach life skills through training on gender sensitive, child friendly, learner directed learning approaches • The capacity of teachers to teach life skills is strengthened • By 2010, the capacity of all life skills teachers in the 10 districts with the highest adolescent HIV incidence rates is strengthened • All teachers in the 10 districts with the highest adolescent HIV incidence rates are able to teach life skills to school children by 2010 • Change Language • Results language to emphasise future condition and take out information relating to strategy or activities • 2. How many? Everywhere? By the year 4000? Be more specific • Bring the subject of change to the front and shift from passive to active language 22

23. To strengthen the capacity of UNICEF staff to do RBP/M by undertaking a workshop… Let’s use results language to emphasise the future condition we want to achieve. The capacity of UNICEF staff isstrengthened to do RBPM by undertaking a workshop… All UNICEF staff everywhere? Can you be more specific? Are there particular staff we should emphasise? The capacity of select UNICEF staff in ICO is strengthened to do RBPM by undertaking a workshop We can take out information that relates to either strategy or activities. The capacity of select UNICEF staff in in ICO is strengthened to do RBP/M by undertaking a workshop Now, let’s try bringing the subject of change to the front, and shifting from passive to active language and add time frame. Select UNICEF staff in in ICO have the capacity to do RBPM by December 2009. 23

24. Quality Criteria for a Results statement • Is it stated using change language? • Is the scale/scope realistically within the control of you and your partners? • Is there an If-then causality between output and outcome and are sum of outputs sufficient to achieve higher level outcome? • Make sure output does not repeat the outcome statement and is not a completed activity (training conducted, workshop completed; survey implemented; law reviewed) • Make sure it is SMART – Specific, Measurable, Achievable, Relevant, Time bound • Take reference to strategy out of sentence • To what extent will the most vulnerable/disadvantaged participate in and benefit from the output; • Has the gender dimension been considered? 24

25. Output in the context of human rights How? What do we want to change? Why? Inputs Activities Outputs Outcomes Impact Change in realisation of rights Implementation partners Plugging capacity gaps of DB/ CH instts Improved performance of DB and/ or behavioural change in DB/CH 26

26. CAPACITY ANALYSIS Components Authority Responsibility/ motivation/ will to do Resources Triple A decision-making capability Communication capability 27

27. The challenge of capacity development: Working towards good practice (OECD paper - 2006) Individual level (experience, knowledge & technical skills) Systemic factors, i.e., relationships between the enabling environment, organizations and individuals Organizational level (systems, procedures & rules) Influences by means of incentives it creates Enabling environment (institutional framework, power structure & influence) Successful capacity development requires not only skills & organizational procedures, but also incentives & good governance 28

28. UNDG Capacity Assessment Framework – a Three-Dimensional Cube (2008) 29

29. The UN System and Capacity Development Background Paper (2006) • Don’t rush- long-term process. • Respect the value systems and foster self-esteem- self-esteem is at the root of capacity and empowerment. • Scan locally and globally; reinvent locally- there are no blueprints. • Challenge mindsets and power differentials- capacity development is not power neutral. • Think and act in terms of sustainable capacity outcomes. 30

30. The UN System and Capacity Development Background Paper (2006)… • Establish positive incentives. • Integrate external inputs into national priorities, processes and systems - external inputs need to correspond to real demand • Build on existing capacities rather than creating new ones- this implies the use of national expertise as prime option. • Stay engaged under difficult circumstances -the weaker the capacity the greater the need. • Remain accountable to ultimate claim holders- any responsible government is answerable to its people, and should foster transparency as the foremost instrument of public accountability. 31

31. Examples of Indicators Goal Impact indicators Infant mortality rate, under 5 mortality rate Objectives results Results monitoring Purpose Outcome indicators Number & % of child diarrhea cases treated with ORT Number & % of mothers knowing about & having access to ORT services Outputs Output indicators Activities Process indicators Number of health personnel trained in ORT, number of media ads produced, etc Strategies means Implementation monitoring Funding amounts ($), trainers (person-months), ORT supplies (numbers), etc Inputs Input indicators Project Logical Framework

32. Business Counseling Example Improved business performance These owners gain knowledge and change practices resulting in A variety of educational activities are provided to business owners who participate Agency invests time and resources 33 University of Wisconsin-Extension, Program Development and Evaluation

33. Example: Water quality 34 University of Wisconsin-Extension, Program Development and Evaluation

34. PROGRAM LEVEL RESULTS CHAIN ATTRIBUTION MacroLong termLow Millennium Development Goals Impacts Country Program Results Outcomes Financial, Knowledge, Sector Results Outputs Budget Resources & Projects Inputs Micro Short term High

35. 36

36. Definitions Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended. Impact Outcome The likely or achieved short-term and medium-term effects of an intervention’s outputs. Output The products and services which result from the completion of activities within a development intervention.

37. Result Chain: Definitions the overall and long-term effect of an intervention; it is the ultimate improvement/ changes in people’s lives;intended/ unintended e.g. Improved access to justicernance Institutional and behavioral changes in the real world triggered by the outputs; outcomes are not under the control of the project any more (in contrast to outputs)e.g. Adherence to international human rights instruments and treaties. Tangible, time-bound products resulting from completion of activities; are under the control of the project e.g. Policy makers, CSOs and judiciary officials trained and able to apply human rights treaties and instruments activities are the various steps taken to carry out a projecte.g. Material on raising awareness on human rights developed and translated into local languages Organize training workshops inputs are means mobilized to carry out activitiese.g. funds, staff time and other resources

38. Basic Services Action Statements: 1. Increase participation of vulnerable groups & their access to basic health 2. Enhance safe and equal access to quality education, health 3. Increase the safe and equal access to primary and secondary education Environment Action Statements: 1. Strengthen capacity to comply with EU/UN standards and environmental conventions 2. Support government/CSOs to develop and implement sound agricultural and environ. policies 3. Ensure sustainable use of natural resources Good Governance Action Statements: 1. Support people-oriented governance through decentralization and local capacity building 2. Build the capacity of government structures to be transparent and accountable Outcome Formulation: An example… There is increased and equal access to quality basic health and education services National capacities for sustainable management and protection of natural resources are strengthened National and local capacitiesfor transparent and accountable provision of decentralized services are strengthened

39. Types of Change Changes in the lives of people Impact Outcome Institutional Change: values, ethic, rules, laws – associated with to institutional performance, access.. Behavioural change: knowledge, skills, practices (individual level) Output Operational Change: provision of goods and services

40. To achieve results you need to.. Set strategic intents which provide a focus for action Formulate “SMART” results Identify baselines Set intermediary targets Select performance indicators Be determined to do it.

41. Selection Criteria for Results S Specific M Measurable A Achievable R Relevant T Time bound & Trackable

42. more Impact Human! 5-10 yrs Collective Accountability Outcome Institutional/ Behavioural 5 yrs Output Operational/ skills, abilities, products & services <5 yrs less Activity <1 yr A Typology for RBM: Poverty Reduction (2) Results Like… Focus @ Timeframe then Poverty reduced if then Outcome Employment and income generation increased Institutional/ Behavioural 5 yrs if then 3000 new small enterprises developed in poorest provinces if then 7 model business incubators operational in poorest provinces if • Acquire facilities • Staff training • Micro-credit provision..

43. Annual Workplan

44. Harmonised AWP Monitoring Tool • Regional (or Country) Programme Component: _________________ • Implementing Partners: _________________Year: ____

45. Result Chain

46. S M A R T What are Good Indicators? SPECIFIC • In terms of quantity, quality, time, location, target groups, baseline and target for the indicator MEASURABLE • Will the indicator show desirable change? • Is it a reliable and clear measure of results? • Is it sensitive to changes in policies & programmes? • Do stakeholders agree on exactly what to measure? ACHIEVABLE • Are the result (s) realistic and based on risk assessment, partnership strategy and other factors contributing to the underlying result RELEVANT • Is it relevant to the intended result? • Does it reflect the expectations and success criteria for change in the target groups? TRACKABLE/TIMEBOUND • Are data actually available at reasonable cost & effort? • Can proxy indicators be used? • Are data sources known?

47. What are Indicators? In other words …. Means of measuring what actually happens against what has been planned in terms of quantity, quality and timeliness, for every level of result. A quantitative or qualitative variable that allows the verification of changes produced by a development intervention relative to what was planned (UNDG Harmonized Terminology, 2003)

48. Indicators ‘indicate’ that change is happening or not happening. Purpose of Indicators They can…. • Clarify the scale and scope of a result in the results framework • Demonstrate progress when things go right • Provide early warning when things go wrong • Assist in identifying changes that need to be made in strategy and practice • Inform decision making • Facilitate effective evaluation

49. Baseline A baseline is the situation just before, or at the outset of a new program, project, service or operation against which progress can be measure or comparisons can be made as part of monitoring and evaluation. Effective monitoring is nearly impossible without an established baseline.

50. Targets A target is an explicit statement or result derived for an indicator over any specified time period (to be provided at the level of output, outcomes and impact).