EDRMS # 6284067

1. EDRMS # 6284067 CIDA’s Approach to Results-Based Management in Fragile and Conflict-Affected States (FCAS) Jennifer Mullowney November 26, 2012 Bangkok, Thailand

2. Presentation Agenda • PART I: Setting the Stage • What is RBM? • 6 key elements • How is it different? • Results Chain Terminology • CIDA’s RBM Tools: Logic Model • (LM), Performance Management • Framework (PMF) & Risk Register • (RR) • PART II: International Landscape of • Engaging in Fragile States • OECD-DAC Principles for Good Engagement in Fragile States • New Deal • Part III: Applying RBM in FCAS • What is the difference? • Theory of Change/Results Chain • ‘DO NO HARM’ as a guiding principle • Conflict Sensitive Indicators • Baseline • Data Collection • Monitoring & Evaluation • Attribution • Part IV: Lessons Learned • CIDA Past Programming Constraints & Successes • OECD EvalNet Summary Report

3. Part I: Setting the Stage What IS RBM AT CIDA?

4. Modern management methodology that requires practitioners to look beyond activities and outputs to focus on: actual results

5. CIDA’s perspective: Results-Based Management • Defining realistic expected results, based on appropriate analyses; • Clearly identifying program beneficiaries and designing programs to meet their needs; • Monitoring progress towards results and resources consumed, with the use of appropriate indicators; • Identifying and managing risks, while bearing in mind expected results and the necessary resources; • Increasing knowledge by learning lessons and integrating them into decisions; and • Reporting on results achieved and the resources involved Source: CIDA’s RBM Policy 2008

6. CIDA’s Perspective: What is a Results Chain? A visual depiction of the causal or logical relationships between inputs, activities, outputs, and the outcomes of a given policy, program, or project. Source: Results-based Management Lexicon, TBS,2004 (Adopted by CIDA in 2008 – Results-based Management in CIDA – Policy Statement) 6

7. DFID 2011 CIDA (1996-2008) TBS & CIDA (2008) NORAD 2008 OECD AusAID Differing RBM terminology used around the world Impact Ultimate Outcome Goal/Impact Impact Impact Impact Outcome Intermediate Outcome Purpose/ Outcome Outcomes (Medium-term) Outcome (Short-term) Outcome Output Immediate Outcome Component Objectives Output Output Output Output Outputs Activities Activities Activities Activities Activities Process The important thing is to understand the concept of causality and not be distracted by the range of terminology.

8. Source: Adapted from St. Mary’s University CIDA’s Results Chain Chang in State or Conditions for ultimate beneficiaries Ultimate Outcome Economic Cultural Social Civic Environmental Change in Behavior, Practice or Performance of intermediaries or beneficiaries Intermediate Outcome Decision-making Effectiveness Efficiency Practice Social Action Behavior Policy making Viability Change in Capacities of intermediaries or beneficiaries Immediate Outcome Knowledge Attitudes Processes Skills Motivations Awareness Aspirations Opinions Products and Services stemming from planned activities of the project/program Outputs Workshop facilitated Policy advice provided Assessments conduced Clinics built or refurbished Training provided Research undertaken Report written Planned activities of the project/program Facilitate workshops Activities Provide police guidance Assess Refurbish infrastructure Train staff Conduct research What we invest Input Technology Money Time Materials Staff Equipment Partners

9. The Logic Model ULTIMATE OUTCOME Improved health for women, men and children living in community X Increased usage of safe drinking water by women, men and children in community X. Improved management of water, waste and sanitation infrastructure in community X INTERMEDIATE OUTCOMES Increased equitable access to safe drinking water for women, men, girls and boys living in community X Increased awareness of the importance and proper usage of safe drinking water among women, men, girls and boys in community X Increased ability of women in community X to maintain wells Increased skills in waste management and sanitation among female and male workers of Regional Water Authority IMMEDIATE OUTCOMES OUTPUTS - Wells built in community X, in consultation with local stakeholders, especially women as primary water managers in the community - Existing wells rehabilitated in community X - Awareness material, including material appropriate for a non-literate audience, developed in consultation with male and female community members  - Awareness campaigns conducted on the importance and proper usage of safe drinking water for women, men, girls and boys in community X Training provided to women in community X on maintenance of wells TA in waste management and sanitation provided to local community development committees members both women and men. - Build wells in community X - Rehabilitate existing wells in Community X - Develop Awareness Material - Conduct Awareness Campaigns Provide training to women on maintenance of wells Provide TA in waste mgt & sanitation to local community development committees ACTIVITIES

10. The Performance Measurement Framework

11. The Risk Register

12. Part II: International Landscape of Engaging in Fragile States

13. Characteristics of a FCAS • Different organizations use different parameters to judge fragility. For CIDA: • A state is considered to be fragile when its government does not demonstrate the will and/or the capacity to deliver core state functions. • A state is considered to have failed when there is no longer a legitimate central authority or regime animating state structures and functions • States are not only fragile when they are declining or moving towards failure, but also when they are recovering from failure. • Fragile states are the furthest away from achieving the MDGs • 1.5 billion people live in conflict-affected and fragile states. • About 70% of fragile states have seen conflict since 1989. • Basic governance transformations may take 20-40 years. • 30% of Official Development Assistance (ODA) is spent in fragile and conflict-affected contexts. (Source: OECD-INCAF)

14. Humanitarian Action • Humanitarian action includes the protection of civilians and of those no longer taking part in hostilities. • It also includes the provision of food, water, sanitation, shelter, health services and other items of assistance for the benefit of affected people to facilitate the return to normal lives and livelihoods. • Humanitarian aid encompasses assistance, relief and protection operations to save and preserve life in humanitarian crises or their immediate aftermath. Source: Good Humanitarian Donorship

15. OECD Principles for Good International Engagement in Fragile States and Situations • Take context as the starting point • Ensure all activities do no harm • Focus on state building as the central objective • Prioritise prevention • Recognize the links between political, security and development objectives • Promote no discrimination as a basis for inclusive and stable societies • Align with local priorities in different ways and different contexts • Agree on practical co-ordination mechanisms between international actors • Act fast, but stay engaged long enough to give success a chance • Avoid pocket of exclusion (i.e. aid orphans)

16. Theory of Change: ‘DO NO HARM’ • A key issue for the design stage of a project is to ensure that donors & partners ‘DO NO HARM’ in their interventions • The most obvious way we may do harm is if our inputs and/or activities (projects, policies or positions) increase conflict or the potential for conflict) • All interventions in FCAS are potentially harmful: • Aid can undermine the structures and relationships in society that enable people to live together (connectors), or conversely it scan enhance the factors that push people apart (dividers) • Aid can provide resources which are redirected for war, or which free up other resources for war, or which increase inequalities (resource transfers) • Aid can legitimise the values of war and positions of warmongers (implicit ethical messages) • Source: Mary Anderson, Do No Harm: How aid can support peace – or war (1999)

17. Planning for Results in FCAS • Identify/Validate the priorities for transition support • Keep priorities simple • Identify and target the sectors that matter most from a peace building and state building perspective • Focus on and fund core peace drivers/conflict mitigating factors • At the Fourth High Level Forum on Aid Effectiveness in Busan (Nov 2011), a number of countries and international organisations endorsed an agreement on a new global direction for engagement with fragile states (NEW DEAL) • The New Deal has agreed to use five key peace-building and state building goals (PSGs) as an important foundation to enable progress towards the MDGs (continued..)

18. The New Deal: Peace building and Statebuilding Goals (PSGs) • Legitimate Politics: Foster inclusive political settlements and conflict resolution • Security: Establish and strengthen people’s security • Justice: Address injustices and increase people’s access to justice • Economic Foundations: generate employment and improve livelihoods • Revenues and fair services: manage revenues and build capacity for accountable and fair social service delivery Source: OECD-DAC, International Dialogue on Peace building and Statebuilding

19. Part III: Applying results-based management in fragile state and conflict affected states (FCAS)

20. RBM in FCAS: What’s the difference? • Expected results, and the activities required to achieve them, are affected to a greater extent by unstable and changing environments. • The means to achieving results are more difficult and unpredictable in FCAS; therefore necessitating more frequent monitoring and greater ongoing adjustments (i.e. corrective action). • There is not necessarily a linear progression from activities to outcomes. • The processes involved are more iterative. • There is a greater premium on more accurate and real-time data for decision-making.

21. Prepared by the Europe, Middle East & Maghreb Branch, Canadian International Development Agency, in collaboration with Universalia Management Group. RBM Application Illustration: Theory vs. Practice Outcomes/Impact (often takes longer than planned) Outcomes/Impact Project Implementation Plan Project Implementation Plan

22. THE Logic Model

23. Theory of Change • Peace and stability are elusive concepts made up of innumerable factors • While not a panacea, theories of change can improve the effectiveness and quality of an intervention • A Theory of change clearly articulates the underlying assumptions and provides a blueprint of the building blocks need to achieve long-term outcomes • A well-defined theory of change offers a clear picture of how an given intervention is expected to work. • In its simplest form a theory of change can be stated as, “ we believe that by doing ‘X’ activities, it will achieve ‘Y’ progress towards results. • Peace building and humanitarian assistance strive to facilitate positive changes (including preventing negative changes) for the beneficiaries, intermediaries, or environment in which they belong to. • No static social moment in time, a change is a continuum from one point to another.

24. How does the theory of change apply to CIDA and programming in Fragile and Conflict Affected States? When developing/validating project design, the logic model should be constructed using a sound theory of change derived from a conflict analysis and any existing pre-conceptual models.

25. Theory of Change: Conflict Analysis A good and up-to-date conflict analysis is essential to reducing harm: • Conflict analysis must address the influence of aid on conflict and fragility, looking in particular at the political impact and the perception of key actors. • Conflict analysis should consider potential negative effects as well as the risks of the project on the conflict. • Conflict analysis should consider mitigation activities to address conflict-related risks. • The depth and scope of a conflict analysis should be appropriate to the existing or anticipated intervention and the conducting organization’s capacity.

26. Theory of Change: Example of Conflict Analysis • Conflict Background • Map of area • Brief country description • Outline of the history conflict 2. Conflict Parties • Identify core conflict parties • What are the relationships between conflict parties? • Behaviour of parties • Peacemaking activities • What efforts have been made in the past/present to resolve conflict? • Who else in the area has a role in peacemaking efforts? • Are you aware of other organizations with whom you might liaise? • National, Regional and State Level Context • Is the nature of the state contested? • How do relations with neighboring states and societies affect the conflict? • What external factors fuel the conflict? • What may change them? Source: UNITAR-POCI Peacekeeping and International Conflict Resolution, 2000

27. Source: Banyan Global, Enterprise Development, Poverty and Conflict, 2010

28. Theory of Change: Peace-building Participatory conflict-analysis is instrumental in identifying the types of changes that are needed to transform conflict. Almost all peace-building initiatives implicitly target the four dimensions of conflict transformation: the personal, the relational, the structural and the cultural dimensions. Source: Lederach J.P., Reflective Peace building: A Planning, Monitoring and Learning Toolkit, 2007

29. Source: Church. C and Mark M. Rogers, Designing For Results: Integrating Monitoring and Evaluation in Conflict Transformation Program, Search for Common Ground

30. Theory of Change: Pre-conceptual Model in Humanitarian Action Source: UNICEF, State of the World’s Children, 1998

31. Theory of Change: Principles of Humanitarian Action Humanitarian Principles Humanity – the centrality of saving lives and alleviating suffering wherever it is found Impartiality – the implementation of actions solely on the basis of need without discrimination between or within the affected populations Neutrality – humanitarian action must not favour any side in an armed conflict or other dispute where such action is carried out Independence – the autonomy of humanitarian objectives from the political, economic, military, or other objectives that any actor may hold with regards to areas where humanitarian action is being implemented Source: Principles and Practices of Good Humanitarian Donorship

32. Results Chain in FCAS • A common misconception is that the ultimate outcome (impact)for a fragile state will sit at a lower level then for an ultimate outcome of a traditional development project. • The ultimate outcome should be set so that is ambitious enough to have a lasting impact on the intervention’s beneficiaries but realistic enough to be achievable given current realities and constraints in the operating environment.

33. THE Performance Management Framework

34. Conflict Sensitive Indicators • Indicators are tools. They are neutral quantitative or qualitative units of measure and analysis that specify what is to be measured along a scale or dimension. • They do not indicate direction. Actual Results do. • Good indicators help manage the project for better results by providing evidence that change is or not happening, so that Managers can take timely correction action during the life of a project • Unit of measure: quantitative or qualitative dimension of change expressed as a number, percentage, ratio, level, scale, proportion • Unit of analysis: what is to be measured over a continuum • Context • Target population for proportionality • Timeframe** (where relevant)

35. Conflict Sensitive Indicators: Best Practices • Begin by considering the data sources • Reassess indicator choices after baselines have been set in order to check that adequate data will be available • Include both indicators of drivers of conflict and fragility (e.g. # of human rights violations) • Include measures of successful peace-building and state-building (e.g. # of ex-combatants reintegrated into their communities) • Include metrics of cost-effectiveness for humanitarian assistance interventions (e.g. # of deaths averted or lives saved) • Use direct indicator s to the extent possible. • Use proxy indicators (indirect measures) when monitoring intangible qualities (degree/level of trust or confidence among groups) or when a direct indicator is unavailable or unfeasible to collect

36. Conflict Sensitive Indicators: Performance Indicator Structure Unit of measure: Unit of analysis: Context: # / Total (numerator / denominator) Deaths (gender/region/ethnic group) From armed conflict % / Total (numerator / denominator) Minority groups (religion/region/ethnic group) Voting in elections Level of (1-5 scale) Factional language Used by elites

37. Conducting a baseline in a FCAS • A baseline is the first critical measurement of an indicator and indicates where expected outcomes stand when an initiative begins • Establishing a baseline should be no different in a fragile or conflict-affected country; however, • Collecting reliable data in fragile contexts is more difficult due to the scarcity of qualified staff and cost of collecting data in insecure environments. • A good baseline study should determine what is measurable and what is the initiative’s realistic possible impact in a fragile situation. • When reliable hard figures are not available; the baseline can be anecdotal, a description of ‘wherewe are now’

38. Baseline: Key Questions for Building Indicators and Baseline Information • What are the sources of data? • What are the data collection methods? • Who will collect the data? • How often will the data be collected? • What is the cost and difficulty to collect the data? • Who will analyze the data? • Who will report the data? • Who will use the data?

39. Data Collection Methods in Insecure Environments Conversations with concerned individuals Key Informant Interviews Census Reviews of Official Records One-time Survey Direct Observation Panel Surveys Community Interviews Participant Observation Focus Groups Field Experiments Field Visits Questionnaires Informal and less-structured methods Formal and more-structured methods Source: World Bank: Ten Steps to Results-Based Monitoring and Evaluation System

40. Risk profile & risk register

41. CIDA’s Key Risk Management Principles • Risk Level: greater risk tolerance is required by management. Higher risks lead to greater results. • Key Risks: more attention given to highest and most sensitive risks. Proper risk definitions allow for more relevant risk responses. • Increased Frequency: more frequent updates and monitoring of the Investment Risk Register and Program Risk Profile. • Flexibility: integrate greater flexibility into risk responses. • Interconnectivity: cognizant that risks are interconnected to keep a holistic approach to risk management.

42. Risk Management in FCAS • Proactive and systematic risk planning should be integrated into all project activities to manage risk effectively. • A comprehensive understanding of risk and the expected results it potentially affects enables a more informed decision-making process. • The key difference is the level of risk and the degree of attention that must be paid to the situation due to a higher level of uncertainty. • More frequent updates and monitoring of the Country Program Risk Profile and the Investment Risk Register Tool is recommended to better adapt to changing situations.

43. CIDA’s Key Risk Management Principles • Setting Realistic Expected Outcomes: setting realistic expected outcomes can help better contextualize development work and responses to risk. • Collaboration and Risk Sharing within CIDA or with other international aid agencies/partners can alleviate stress on individual programs. • Analysis: scenario analysis can present potential future event that may affected expected outcomes. • Exit Strategy: preparing a contingency plan and coordinating its application with other federal ministerial departments can be an effective risk management measure. • Diversified Tools: use of diversified risk tools can assist CIDA programs’ response capacity to risk events

44. Monitoring & evaluation

45. Monitoring in FCAS • Results-based monitoring is a continuous process of collecting and analyzing information on indicator in order to measure progress towards expected outcomes. • Whereas traditional monitoring focused on inputs, activities and outputs (project implementation) results-based monitoring combines the traditional approach of monitoring implementation with the assessment of results. • Conflict-Sensitive monitoring builds on Results-based monitoring by enabling project staff to gain a detailed understanding of the conflict context, the project outputs and expected results, and the interaction between the two. • Conflict-Sensitive monitoring introduces an understanding of conflict actors, causes and dynamics to inform required adjustments and changes to project/program activities. It is also more sensitive to the need for real-time performance information.

46. Evaluation: International Context • The 2005 Paris Declaration on Aid Effectiveness advocates for partner country ownership, alignment, management for results and mutual accountability • The 2008 Accra Agenda for Action built upon the Paris Declaration through outlining policies and actions that will improve transparency and accountability through strengthening effectiveness • The 4th High Level Forum in Busan reiterated the need for joint evaluations and sharing of lessons from the jointly implemented initiatives in the context of increased partnerships and joint-donor initiatives (e.g., direct budgetary support, pooled funding to sector programs)

47. Evaluation at CIDA:DAC Network on Development Evaluation Evaluation at CIDA: Positioning Evaluating in the Current Context • DAC-EVAL: guidance, norms, quality standards • Standard criteria for development evaluation, adapted for CIDA: • Relevance • Effectiveness • Sustainability • Cross-cutting themes: • Environmental sustainability, Gender& Governance • Coherence • Efficiency • Performance Management 47

48. OECD-DAC Eval-net Principles • Take context as the starting point – the need for conflict analysis • Combine evaluation approaches – not one correct or blueprint approach for undertaking evaluations of donor engagement in FCAS • Conflict sensitivity –doing harm in a situation of conflict and fragility mean having impacts (intended or not) • Gender awareness and equality –field experience and research show that the way women and men experience, engage in and are affected by violent conflict and situations of fragility • Manage expectations – set realistic outcome on what can be achieved and covered in limited timeframe by an evaluation. Be flexible. • Protection and ethical responsibilities – conducting evaluations in areas of conflict and fragility may expose evaluation teams at risk • Independence of the evaluation team • Participation of relevant stakeholders • Transparency of evaluation findings

49. Attribution • It is important to keep in mind that there is currently no clear or error-proof way of assigning attribution for the consolidation of peace to any one particular actor. • Given the complexity of most conflict affected and fragile environments, intervening partners and donors will at best be able to demonstrate that evidence of change coincided with positive changes in the operating environment.

50. Part IV: Lessons Learned from past CIDA PROGRAMMING