COMING OF AGE OF JOINT EVALUATIONS? ALNAP Presentation at OECD-DAC 21 st January 2008

1. COMING OF AGE OF JOINT EVALUATIONS?ALNAP Presentation at OECD-DAC21st January 2008

2. Agenda • Background • Findings • Future Agenda for Humanitarian JEs • Other relevant ALNAP work

3. Background • ALNAP is a network of all the major humanitarian actors • Now in its tenth year • Works to improve humanitarian performance through learning and accountability • Range of products and outputs, including a meta evaluation of humanitarian evaluations • periodic review a sample of evaluation reports against a Quality Proforma which has been developed according to accepted evaluation good practice. • Systematic use of the Proforma over a number of years has made it possible to identify trends in evaluation quality over time.

4. The ALNAP meta-evaluation series • Meta-evaluation defined as • “meta-analysis and other forms of systematic synthesis of evaluations providing the information resources for a continuous improvement of evaluation practice”. • Overall aim of ALNAP meta evaluation is • “...to improve evaluation practice, by identifying areas of weakness that deserve attention, and examples of good practice that can be built upon...” • There is qualitative evidence that this aim is being met • E.g. the use of DAC criteria in EHA has gradually strengthened in the last few years; consultation with primary stakeholders has improved in evaluation methodology (despite still being poorly rated overall) • In other areas there has been little or no improvement, for example attention to cross-cutting issues such as gender equality, protection and advocacy • Where improvement in quality has been noted, it has usually happened quite gradually .

5. Humanitarian Joint Evaluations • In the humanitarian sector, the first significant joint evaluation was the seminal multi-agency Rwanda evaluation in 1996 • shorter history than in the development sector • Subsequent JEs have usually been collaborative efforts of donor governments, but involvement is broadening to include UN and NGOs • The 6th ALNAP Meta Evaluation focuses on growing number of Joint Evaluations • Currently being finalised, findings presented here for discussion and debate

6. 6th ALNAP Meta evaluation: specific objectives • To review the quality of joint evaluation exercises, where possible comparing these with the quality of past single agency evaluations reviewed in previous ALNAP meta-evaluations; • To document in an accessible way the lessons from the growing experience of humanitarian JEs – especially examples of good practice – to feed into future joint endeavours • Thus, to make a significant contribution to the emerging body of knowledge about humanitarian JEs

7. Meta Evaluation Methodology • The methodology used was comparable to previous meta-evaluations, based on a sample of 18 evaluation reports • The quality of the evaluation reports has been assessed against the ALNAP Quality Proforma (slightly adapted to be appropriate to JEs) • http://www.alnap.org/resources/quality_proforma.htm • The evaluation processes have been reviewed through individual and group interviews with those involved in the JEs, iterating between these two methods • Interviews have been carried out with representatives from 15 different organisations as well as lead / central evaluators • The data from the assessment against the Proforma was analysed and compared with the results from previous ALNAP meta-evaluations, which have covered a total of 138 evaluations.

8. Typology of JEs (adapted from DAC, 2005) • ‘Like-minded agencies’ (or qualified): agencies with similar characteristics coming together • WFP/ UNHCR pilot food distribution (UN agencies operating to an MOU) • All ECB evaluations (group of NGOs); • DEC evaluations (group of NGOs); • IASC RTEs (UN agencies) • Most common • ‘Partnership’: donor recipient agencies evaluate together as equal partners • ECHO/ WHO/ DFID JE: WHO emergency response, Pakistan • ‘Hybrid multi-partner’: disparate actors coming together playing variable roles (eg active/ passive) • IHE evaluations (comprising UN agencies, NGOs academics, recipient government etc) • ‘System-wide’: open to all actors in the system • TEC

9. Agenda • Background • Findings • A Future Agenda for Humanitarian JEs • Other relevant ALNAP work

10. Ten Hypotheses to be tested • Humanitarian Joint Evaluations... • help to build trust and social capital within the sector • tend to be driven from the centre (ie headquarters) than the field • do not involve sufficiently the government of the area affected by the humanitarian crisis • offer greater opportunity for beneficiaries to be consulted/ surveyed than in single agency evaluations • have more rigorous methodologies than single agency evaluations

11. Ten Hypotheses to be tested • Humanitarian Joint Evaluations... • pay more attention to international standards and guidelines than single agency evaluations • are stronger on cross-cutting issues such as gender and protection than single agency evaluations • overall quality tends to be higher than single agency evaluations • are more likely to address both policy issues and programme performance than single agency evaluations • pay attention to wider debates within the humanitarian sector, and situate their findings accordingly

12. Findings in relation to hypotheses (1) • JEs are no longer solely the domain of donor governments, the early champions of JEs • UN agencies and some NGOs are now fully engaged • still early days, and some efforts to promote and institutionalise a JE approach have come and gone,despite evidence that JEs help to build trust and social capital amongst the participating organisations (hypothesis 1) • JEs have so far been northern and headquarters-driven (hypothesis 2) • Reflecting the set-up of international humanitarian agencies • Real progress is needed in in fully involving stakeholders in-country - national NGOs, other organisations and governments • Generally poorly represented (hypothesis 3). Involving latter will be easier in natural disasters than in conflict-related humanitarian crises, especially if government is an active party in the conflict • There may be important lessons for the humanitarian sector from JEs on the development side, from work done by DAC to strengthen developing country participation and from the Paris Declaration on Aid Effectiveness

13. Findings in relation to hypotheses (2) • Conclusive evidence that JEs are overall of higher quality. than single agency (hypothesis 8) • their terms of reference are generally clearer and more useable • consultation with local populations and beneficiaries is stronger (hypothesis 4) • more attention is paid to international standards (hypothesis 6); and the OECD-DAC EHA criteria are more rigorously used • JEs have more rigorous methodologies than single agency evaluations is proven, but not across the board (hypothesis 5) • There are striking gaps and weaknesses in the JEs reviewed, especially their attention to cross-cutting issues such as gender equality, protection and advocacy (hypothesis 7). • Hypotheses that JEs are more likely to address policy issues and locate their findings within wider debates within the sector met with a mixed response (hypotheses 9 and 10) • There is some evidence of this (e.g. UN-led humanitarian reform processes), but also missed opportunities, a number of JEs in our sample had not fulfilled this potential, despite the generally high quality

14. Broader findings: most JEs of humanitarian action are multisectoral, focus on a particular humanitarian crisis

15. Broader findings: Purpose of JEs • The ToR for most JEs in the sample emphasise both accountability and learning as their purpose • in practice learning has dominated • ranging from learning about partners’ approaches, to sharing good practice, to learning about the programme or initiative being evaluated • Both the TEC and ECB clearly identified another learning purpose: to learn from the process of implementing a joint evaluation, and buyild evaluation capacity • Accountability is partially met by JE reports ending up in the public domain. Peer accountability is also strong in JEs which usually demand a higher level of transparency than single agency evaluations • Other purposes of JEs include: building evaluation capacity, learning about the process of doing JEs, and relationship-building between participating agencies

16. Broader Findings: JE skillset • JEs require a different skill set than single agency evaluations • technical, political and inter-personal skills • Getting the right evaluation team in place is key • the pool of sufficiently skilled evaluators for JEs is small compared with demand, implying a need to invest in evaluator capacity • For policy-focussed evaluations, there are benefits of hiring policy analysts to lead or be part of the team.

17. Broader Findings: Follow-up & utilisation • Generally more accessible than single agency evaluation reports • Possibly because of the higher skill set of evaluation team leaders • Use of professional report editors may play a role • Conclusions are slightly stronger than evaluations in previous years, but there is little difference in the quality of recommendations • strongest sets of recommendations were those targeted to individual agencies or where responsibility was clearly indicated • the weakest where there were too many recommendations and/ or they were inadequately focussed • Utilisation-focus is more challenging for JEs because of the range of stakeholders involved with different needs • weak link in the chain • there are examples of good practice in terms of how the process is designed at the outset and how the evaluation team engages with stakeholders, especially in-country. • When JEs are part of a wider institutional framework/ relationship, there tend to be better-established mechanisms for discussion and follow-up to recommendations

18. Broader findings: coherence with single agency evaluations • “Should JEs replace or reduce the need for single agency evaluations?” • Active but distracting debate • “they are very different animals” with different purposes • JEs can fulfil accountability purposes, but at a different level needs of a single agency. Accountability to peers, and to some extent to beneficiaries through stronger consultation, are features of a number of the JEs in our set. • But if individual agencies need to be accountable to their funders in any detail, a JE may not fulfil this need. • JEs clearly complement single agency evaluations by placing the response in the bigger picture/ wider context, exploring how agencies work together, and addressing wider policy issues. • when a “club” group of like-minded agencies come together to evaluate their work in a particular area (e.g. ECB / UN) reducing the number of evaluation teams on the ground asking very similar questions of local communities, government officers and others, is clearly a good thing. • Fewer but more considered JEs of this type may facilitate follow-up by reducing the overload on the humanitarian sector

19. Agenda • Background • Findings • A future agenda • Follow up ALNAP work

20. Where next for humanitarian JEs? • Project that analyses and describes the pros and cons of different collaborative management structures, to guide decision-makers in their future choices • Action research project exploring different and creative ways of consulting beneficiaries • JEs in some thematic and policy areas in the humanitarian sector that are relatively new and/ or challenging to the international system • e.g. protection as part of humanitarian action, or livelihood support in the midst of a humanitarian crisis • JEs should play a more active role in agency evaluation policies, based on clear understanding of the relative costs and benefits of different types of JEs

21. Where next for humanitarian JEs? A planned system-wide JE • A 3rd system-wide humanitarian evaluation should be considered in the next 18 months, focussed on a significant but relatively forgotten/ under-evaluated humanitarian crisis, for example in eastern DRC • The sector has much to learn from such an exercise, yet there would be less pressure to act fast at the expense of process, and it would provide an opportunity to apply the learnings from the TEC whilst they are still fresh. This proposal should be discussed by the ALNAP membership and the wider aid community.

22. Agenda • Background • Findings • Future Agenda for Humanitarian JEs • Follow up ALNAP work

23. ALNAP follow up work of relevance to the OECD-DAC Evaluation Network • Workshop on Humanitarian JEs (2nd half 2008) • ALNAP Guide to Real Time Evaluation • end of March 2008 • Humanitarian Evaluation Trends and Issues • initiating case study based research of humanitarian evaluation, background study nearing completion • 7th ALNAP Meta Evaluation (2009) on RTEs

24. Thank you! • 6th ALNAP Meta Evaluation Available in March / April • Please get in touch for copies • Ben Ramalingam b.ramalingam@alnap.org www.alnap.org