Challenges of Joint Peace Intervention Evaluations: Practical Solutions

1. Four Rules of the Art AND SOME POKING AROUND FOR PRACTICAL SOLUTIONS TO THE CHALLENGES OF JOINT PEACE INTERVENTION EVALUATIONS Emery Brusset NUPI/IPI Workshop, New York 7 & 8 May 2009

2. Four Theoretical Challenges to the evaluation of conflict interventions • 1. Seizing Opportunities • Programming requires a response to constraints and opportunities as they arise. • These cannot be planned for in advance and plans are not a good evaluation reference • 2. Attribution • It is particularly difficult to separate the impact of peace-building projects from other impacts on conflict • The time required to cover other interventions is limited • 3. Data Quandaries • Selecting evidence requires prioritisation and demands a good understanding of the nature of the conflict • The verification methods must be applicable in a conflict situation • 4. End State? • Problems in defining terms such as peace and conflict. • Usually depends on the identification of an end-state. • This may not be explicit or shared by all stakeholders or those within a single programme Options Channel Research

3. The Practical Challenge of Joint Evaluation: Evaluation needs to provide a convincing set of conclusions to a growing diversity and size of audience • 4 Rules of Good Evaluation: • Independence of team to focus on situation as it emerged • Team has means to analyse societal interface • Team is capable of participatory assessments • Conflict objectives are reconstructed “Ideal single actor evaluation” “Reality of Joint Evaluation” • TOR enable the evaluator to identify unplanned and unintended effects • A way is found to identify comparative weight of intervention • Good conflict analysis is possible • The nature of “peace” is agreed • Expectations evolve • The questions focus on the management of interventions • There is no strategic baseline or agreed assessment • Peace is defined generically, or not at all to From Channel Research

4. Challenge 1: Capturing unintended and unexpected aspects • Rule 1: The team is enabled to define what issues to focus on, • and allows for Steering Committee consultation milestones • to secure alignment of all stakeholders. Channel Research

5. Assess RISK Ongoing Analysis Rolling Assessment Intensive Management Ongoing Decision Making and Documentation Adapt Act Evaluation, particularly real time evaluation, tends to be linked to decisions and the need to understand and support the management of an intervention. This risks reducing the time available to analyse the impact . Challenge 2: Evaluation questions tend to focus on institutional issues

6. $$ Rule 2: Evaluations are given time to enable an iterative but systematic process (six months to a year?) – scope is contained. There is agreement that attribution of impact is replaced with contribution. Threat Threat Threat Threat ? Threat Success depends on sufficient resources Threat Channel Research

7. Challenge N° 3: Get the readers and audience to agree on a consistent analysis of the conflict, and on key drivers of peace Map of situation Situation with intervention factored in – how does it affect situation, how does situation affect it? Intervention footprint Channel Research

8. Participatory Mapping of Immediate Conflict Environments (MICE) works by looking at linkages between key trends and events = drivers Hypothetical and partial example: Country with military regime in conflict with leftist insurgents The elements of conflict mapping Battle in which war was lost Officers inc. disgruntled Event: occurs at a point in time Coup Corruption more apparent Trend: change in a condition or characteristic Dissent increasingly suppressed Growing leftist radicalisation Arrow of change: increased probability or intensity Many insurgent Attacks in XX Growing Unemployment Channel Research

9. Battle in which war was lost Officers inc. disgruntled Coup Corruption more apparent Dissent increasingly suppressed Growing leftist radicalisation Many insurgent Attacks in XX Growing unemployment The aim is the identification of key issues that need to be addressed to mitigate conflict or instability To identify priority issues, we count the linkages leading to and from relevant events and trends – those with the most linkages have the most reverberation in the situation Priority issues: 1. Leftist radicalisation 2. Militarism, state repression, military dictatorship CAVEAT Conflict mapping is not root causeor systems analysis. Although corruption and unemployment might be major causes of leftist radicalisation, we do not identify them as priority issues – ranking is not intuitive at all 4 4 4 5 Channel Research

10. Representation is qualitative and culturally neutral. For example… Company unpopularity on ground Increasing state revenues Rebel threats to company Military regime corruption Military Politicisation increases _ Leftist perception of collusion Military coup Increasing employment Leftist radicalisation Oil production increases More Poverty Leftist uprising Leftist – peasant cooperation Military steps up project security Peasant revolt US company oil contract _ Shareholder scrutiny Labour activism General strikes Strikes among project staff Human rights NGO scrutiny Military repression Pressure for democratisation _ US gov’t criticism of military regime Channel Research

11. An example from a real mapping exercise in a post-conflict African country Channel Research

12. Table 1: Key ‘nodes’ emerging in different mapping workshops, ranked according to emphasis given in each workshop Mapping of Cross-Border Issues in the Conflict in NI Channel Research

13. Challenge N° 4: objectives and their connection to peace are unclear or outdated Channel Research

15. Complexity in PLANNING is not well captured by prevailing tools “Project” “Strategy” LogFrame ???

16. Rule 3: allow the team to innovate and interpret in mapping objectives and their meaning in terms of peace “Project” “Strategy” LogFrame “Effect Diagram”

17. The intervention tree in its simplest form – it is an adapted logic or decision tree Each new branch needs to be a separate and complete initiative to the extent possible, to ensure no redundancy, and to spot any gaps This This … this, and to see this, we need to do… This We need to achieve this, and to achieve this we need to have obtained… This … this, and to see this, we need to do… This This … this, and to see this, we need to do… This This This Channel Research

18. EC activities Outputs / Outcomes Impacts Health Access to BHCP improved / increased Positive tendency in health status of population (especially U5 and maternal mortality)‏ Health sector management on central and provincial level improved Vulnerability of rural population decreased Rural Development Sustainable management of natural resources achieved / improved Alternative rural livelihoods created Rural production and gainful employment strengthened Rural economy strengthened Poverty reduced Equitable access to productive assets, markets and services (e.g. microfinance) established (Participation of) rural communities in dev. Strengthened More effective provincial / community based institutions developed Poppy production reduced or stabilized State of law and order improved (incl. drug enforcement Public Administration Reform Governmental / admin. structure strengthened National payment system created Example: EC Effect Diagram Afghanistan Civil Service System improved (including police)‏ Foundation for public administration reform created (NOTE: in 2nd NIP, increased focus on sub-national level)‏ Reform of civil service promoted Governmental physical infrastructure re-created Revenue position of Gov strengthened Gov / ATA budget stabilized

19. Actions Outputs Outcomes Direct Impact Global Impact Support measures for regional integration mechanisms and trade co-operation implemented Regional co-operation and integration strengthened Tensions between countries reduced Trade links between countries strengthened. CSP used as central instrument for mainstreaming conflict prevention Macro-economic assistance operations implemented (12). Macro-economic environment in at risk countries stabilized. Structural Stability improved 1) Targeted actions implemented to support electoral process, parliamentary activities, administration of justice (14). Potential for conflicts between partner countries reduced Integrated Approach Rule of Law, Good Governance & Civil Society strengthened & (13) Economic growth stimulated Support to security sector increased (I.e. police training, HR training) (14). Poverty reduced (14). Security sector prepared for post-conflict situation Rehabilitation and reconciliation programmes implemented (16). Potential for more favourable democratic environment increased(14). Potential for renewed conflicts in post-conflict countries reduced (16). Societal post-conflict tension reduced Ownership of process by partner country established • Mechanisms developed to appropriately address cross-cutting issues in relevant areas: • social conditions (drugs, small arms, Health / communicable diseases (HIV / Aids), Population flows / trafficking) • environment & natural resources (water) • economy (Private sector in unstable areas) Cross-cutting issues efficiently addressed in EC activities Peace in post-conflict countries consolidated (16). Conflicts in partner countries prevented Equal participation of men & women in social, economic & political life promoted (14). Potential for new conflicts in at at-risk countries reduced. EU decision-making & operational centres alerted to imminent crisis EC enabled to formulate appropriate response to developing crisis situation (incl. preventive sanctions) Early-warning mechanism & procedures created information on cause & potential consequences of conflict provided Rapid Reaction Mechanism (RRM) introduced EC enabled to carry out wide-ranging short term missions on short notice EC enabled to react quickly to developing crisis situation Co-ordination of Member States on common political line intensified (incl. sanctions)(23). More focused, time-flexible & robust political dialogue & diplomacy enabled (23) Degeneration of political difficulties / tension into armed conflict prevented Reform of aid management allows more responsiveness to needs in line with the performance of partner countries Traditional community instruments optimised to respond to conflict (including ESDP) Coherence of approaches for conflict prevention improved among partners (countries, international organisations, NGOs) Dialogue with partners intensified & focused on conflict prevention (incl. NGOs)(25) Common early warning systems, monitoring procedures & conflict indicators developed Appropriate mechanisms & procedures for coordination with international partners developed. 1) Elements of Structural Stability: sustainable economic development, democracy & respect for Human Rights, viable political structures, healthy environmental & social conditions, capacity to manage change without resorting to conflict. Cooperation with international partners intensified Channel Research

20. Analysis of Effect Diagrams What is their purpose? • Visualize hierarchy of objectives / expected effects • Reveal logical links, help to define certain Evaluation Questions & assess internal coherence of strategy What are their weaknesses? • Do not capture changes over time or feedback loops • “Other actor” / “Implementation” / “Context” not captured Conclusion? • For complex interventions: can be used for sub-diagrams • Shows missing elements • Does not provide the final leap from objective to situation change Channel Research

21. The diagram can be turned into a narrative: theories of change • Theories of change can take the simple format: “We believe that by doing X (action) successfully, we will produce Y (movement towards peace).” • Or the chain of logic might involve multiple steps: “if we do X, that will lead to Y, which will encourage Z, which will exert an influence on key actor A.” • In order to create, uncover or test a theory of change, it is essential to conduct a comprehensive analysis of the conflict, identify the key factors of conflict and the relationship between them, and spell out how the intervention (project, programme or policy) will create change. • Making theories of change explicit is a central part of planning and programme design and can facilitate evaluation. Channel Research

22. List of Identifiable Theories of Change: • The Individual Change Theory: Peace comes through transformative change of a critical mass of individuals, their consciousness, attitudes, behaviors and skills. [Methods: investment in individual change through training, personal transformation/consciousness-raising workshops or processes; dialogues and encounter groups; trauma healing.] • The Healthy Relationships and Connections Theory: Peace emerges out of a process of breaking down isolation, polarization, division, prejudice and stereotypes between/among groups. Strong relationships are a necessary ingredient for peacebuilding. [Methods: processes of inter-group dialogue; networking; relationship-building processes; joint efforts and practical programs on substantive problems.] • The Withdrawal of the Resources for War Theory: Wars require vast amounts of material (weapons, supplies, transport, etc.) and human capital. If we can interrupt the supply of people and goods to the war making system, it will collapse and peace will break out. [Methods: anti-war campaigns to cut off funds/national budgets; conscientious objection and/or resistance to military service; international arms control; arms (and other) embargoes and boycotts.] Channel Research

23. The Reduction of Violence Theory: Peace will result as we reduce the levels of violence perpetrated by combatants or their representatives. [Methods: cease-fires, creation of zones of peace, withdrawal/retreat from direct engagement, introduction of peacekeeping forces/interposition, observation missions, accompaniment efforts, promotion of nonviolent methods for achieving political/social/economic ends.] • The Root Causes/Justice Theory: We can achieve peace by addressing the underlying issues of injustice, oppression/exploitation, threats to identity and security, and peoples’ sense of injury/victimization. [Methods: long-term campaigns for social and structural change, truth and reconciliation; changes in social institutions, laws, regulations, and economic systems.] • The Institutional Development Theory: Peace is secured by establishing stable/reliable social institutions that guarantee democracy, equity, justice, and fair allocation of resources. [Methods: new constitutional and governance arrangements/entities; development of human rights, rule of law, anti-corruption; establishment of democratic/equitable economic structures; economic development; democratization.] • The Political Elites Theory: Peace comes when it is in the interest of political (and other) leaders to take the necessary steps. Peacebuilding efforts must change the political calculus of key leaders and groups. [Methods: raise the costs and reduce the benefits for political elites of continuing war and increase the incentives for peace; engage active and influential constituencies in favor of peace; withdraw international support/funding for warring parties.] Channel Research

24. The Grassroots Mobilization Theory: “When the people lead, the leaders will follow.” If we mobilize enough opposition to war, political leaders will have to pay attention. [Methods: mobilize grassroots groups to either oppose war or to advocate for positive action. Use of the media; nonviolent direct action campaigns; education/mobilization effort; organizing advocacy groups; dramatic events to raise consciousness.] • The Economics Theory: As a politician once said, “It’s the economy, stupid!” People make personal decisions, and decision-makers make policy decisions based on a system of rewards and incentives and punishment/sanctions that are essentially economic in nature. If we can change the economies associated with war-making, we can bring peace [Methods: use of government or financial institutions to change supply and demand dynamics; control incentive and reward systems; boycotts.] • The Public Attitudes Theory: War and violence are partly motivated by prejudice, mispercep­tions, and intolerance of difference. We can promote peace by using the media (television and radio) to change public attitudes and build greater tolerance in society. [Methods: TV and radio programs that promote tolerance; modeling tolerant behavior; symbolic acts of solidarity/unity; dialogues among groups in conflict—with subsequent publicity.] Channel Research

25. Families PCs Banks Community Leaders MoH Police State NGO State NGO State NGO State NGO State NGO State NGO OTI+I.P. mission Outcome Mapping: Focusing on actors and behaviour OTI HQ

26. Challenge N°5: Bringing it all back together The Significance of Impact methodology has been designed to provide solid evidence on the contribution which an actor makes to a particular issue Focus of evaluation Outcomes Relevance x Extent x Duration Issue 1 Linkage mapping Outcomes E T E Issue 2 T E T Outcomes T E Issue 3 Outcomes Channel Research

27. Issues identified with the mapping tools can provide the starting point • Priority issues • Poor relations with local community • Rivalry between central and provincial administration • Criminal economy • Ethnic cleavages lead to growing fear of threats • Extent and Duration • What proportion of the target population is covered? • Were the key people accessed? • How long did the influence last? • Can the influence be sustained after the ending of the intervention? Priority risks a d Extent SIGNIFICANCE c b Duration

28. Leading to three dimensional assessments… C. . . D. . . • Priority issues • Poor relations with local community • Rivalry between central and provincial administration • Criminal economy • Ethnic cleavages lead to growing fear of threats • Extent and Duration • What proportion of the target population is covered? • Were the key people accessed? • How long did the influence last? • Can the influence be sustained after the ending of the intervention? • Relevance: • What do others do? • Are there other options? • Could a clearer objective have been designed? • Does the intervention correspond to the need assessed? Degree of relevance B. . . A. . . Extent and Duration D C Significance Extent B A Duration The tools is designed to highlight what has been done and test its connection to the issues which are recognised as issues to be addressed to achieve ‘progress’

29. All levels of the result chain can yield important results or outcomes: demobilisation Outcome Strategy Activities and process deliver output Outputs are defined and easily verified Example: “a workshop is held” Process can be the output The issue can be that groups have lost contact and need to build mutual respect Example: “Familiarity is achieved”. First level Operational level Specific objectives are the results of activities and ideally are how outputs are to be used by the beneficiary groups – user interface Example: “skills and tools are taught” New capacities can lead to changes in the command structure and the reintegration of groups. Example: “Specialisation and opportunities are required for conversion to civilian life” Second level Programme level General objectives are the statement of overall verifiable change intended Example: “Armed groups are able to design a new force concept” Changes in the overall environment can lead to a better notion of security based on smaller forces Example: “Security imperatives require larger armed groups”. Third Strategy Level Society & State Channel Research

30. The Table of Contents of the Evaluation Report! • Issues / Entry Points have been identified • Outcomes achieved have been identified • Assessment of the value of the contribution is possible: • Relevance and efficiency • Effectiveness • Sustainability • Coherence with other interventions • Overall conclusions on « posture » • Recommendations Channel Research