Improving Evaluation Quality: Strengths and Limitations of EQS

1. Session 2 Exploring the Issues Group Work

2. Exploring the Issues - Retrospective Group Work: To what extent have the standards led to better quality evaluations, in particular.. What have been the strengths & success in using the EQS? What have been the limitations & expectations of the EQS?

3. EQS The French speaking group Justine, M, Modibo Makalou, Gerhard Siegfried, Benoît Chervalier, Dominique de Crombrugghe

4. EQS strenghts Used at several steps of evaluation: • Reference framework, also with other standards (eg SEVAL) • Selection of consultants • Report reading tool : grid in 10 points (Fr); quality grid in BMZ, GTZ, KfW • Makes comparative reading of reports possible and helps capitalise on consultants’ work

5. EQS Strenghts (continued) • Helps overcoming desks’ resistance on evaluations • Builds confidence with partners • Helps convincing policy level • Reference framework for meta evaluations

6. EQS Weaknesses • Not yet systematically incorporated in tor • Uneasy access for operational desks • Dissemination and training on ESQ insufficient (both for evaluators and decision makers) • Standards not known (Senegal, Benin) • Failing standards, evaluation design and implementation is scattered • Use of standards, among other tools such as guidances, still confused

7. EQS limitations • Utility of evaluations: good standards are not a sufficient condition for useful evaluations • There is littel feed back by consultants on the standards • Partner appropriation of standards still weak • Some rules are still difficullt to accept in other cultural environments such as disenting opinions • They are donor standards

8. EQS Attention points • Make it clear who the users are • Training and dissemination on the standards

9. DAC EVALUATIONS QUALITY STANDARDS WORKSHOP GROUP ONE (1) PRESENTATION

10. Background: • Mix of Players in the group • Some have used and others have not • Recognition of importance of M&E • Other Standards – UNEG, Bilateral, IFIs • Viet Nam further ahead

11. Strengths and Successes in using EQS??? • Some common core items help to avoid multiplicity of requirements • Facilitates donor/partner country efforts • Some Partner’s EQS (tools, techniques, standards) must be built, strengthened and sustained • Current EQS forming the basis of Uniform EQS (must be encouraged) • Joint Evaluations must be encouraged (harmonisation) • Potential use at front end to encourage results management • Potential for professionalisation

12. Limitations and Expectations with the EQS??? • Long and can be reduced/simplified • Standards are mixed of guidance and quality criteria (need to determine focus) • Methodological aspects can be improved • Cultural context and mechanism for feedback not adequately covered • Paris Declaration considerations are not covered • Ethical aspects are not adequately covered • Challenges for smaller partner countries - capacity, data availability and quality problematic

13. Group 2:Widjanarko Soebhadi, Miranda Cahn, Ginny Chapman, Ted Kliest, Gunila Tornquist Strengths -Informs how people should work -Consistent standards across countries -Can assist in development of country standards -DAC provides credibility -Improves probability of higher quality evaluations

14. Recommendations • Stress and institutionalize constructive nature of evaluation • Emphasize partnership with partner country • Evaluations should cover role of both donor/partner • Emphasize capacity building of partners • Need separate section on management and governance (6.1-8.2) • Include but do not specify cross cutting issues • Review Methodology section for level of specificity and clarity • Rearrange so as to link the sections on objectives to evaluation questions • Better promotion, awareness and dissemination of EQS

15. Group 3 • To what extent have the EQS led to better quality evaluations, in particular? • maybe, but it is hard to tell as implementation is occurring progressively • help to develop a common language and shared understanding • potential to develop increased co-ordination

16. What have been the strengthens and success in using the EQS? • do seem to be providing a basis for shared understanding and potential for better co-ordination • successes are not the same for donors and partner counties • self-reflective tool - this seems to be leading to clarity for improvement areas • treated as flexible therefore allow for the needs of individual partner and agency variation

17. What have been the limitations and expectations with the EQS? • Donor focused - Language and emphasis - written from a donor perspective – e.g. Refers to AID policy not development policy • Partner country view - the EQS aren’t well know, including across a range of Departments - some have their own standards, therefore EQS is in addition (local systems, including evaluation, need to be recognised) • EQS might be too ambitious already – will the EQS become too complex • might be simplified to be more principle-based • Capacity building

18. Group 4Evaluation Quality Standards + - Part of bigger picture Logic missing – structure could be change to cover: design; conduct; product. Dissemination • Generic • Flexible • Checklist • Useful - Developing evaluation policy