DAC Evaluation Quality Standards Workshop, Auckland 6/2 & 7/2 2009

1. DAC Evaluation Quality Standards Workshop, Auckland 6/2 & 7/2 2009 Evaluation quality standards in Dutch Development Cooperation Ted Kliest Policy and Operations Evaluation Department Netherlands Ministry of Foreign Affairs

2. Evaluation quality standards in Dutch Development Cooperation Subjects: • Organisation of evaluation: central & decentral • Development of quality standards from the mid-1990s onwards • Use of standards to improve evaluation quality (ex-ante) and assess quality of evaluations (ex-post) • DAC Evaluation Quality Standards: some observations

3. 1. Organisation of evaluation in Dutch development cooperation • MoFA: a dual system of central and decentral evaluations • Central level = independent Policy and Operations Evaluation Department (IOB): • Policy, thematic and (country) programme evaluations, and recently impact evaluations • Increase in joint evaluations • Guidelines for IOB-evaluations (= methods and procedures) • Decentral level = evaluations commissioned by operational units and embassies (mainly programmes and projects) • Dutch development NGOs have own evaluation functions

4. 2. Decentral evaluation at MoFA: the need for standards • Starting in the ‘90s, IOB conducted periodic reviews of the quality (and utility) of decentral evaluations • Quality criteria developed for these reviews • Periodic reviews led to a Guidance for Decentral Evaluations to be applied in the MoFA • Guidance includes quality standards & checklist to foster quality evaluations (ex-ante) and assess qualityof evaluations (ex-post) • Quality standards and checklist used to assess quality of programme evaluations of Dutch NGOs • Experience with standards used by IOB when involved in developing the DAC Quality Standards

5. 3. Use of standards checklist to assess evaluation quality • Checklist focuses on three main criteria: • Validity - did the evaluation measure what was intended to be measured? • Reliability - are the evaluation findings dependable? • Usability - can the evaluation results be used? • It includes a clarification of concepts • It includes criteria & indicators for assessment • Quality is scored by means of a 4-point rating system: ‘poor’, ‘mediocre’, ‘satisfactory’ and ‘good’

6. Example: application of ‘Validity’ in the evaluation assessment form Validity operationalised in 4 criteria: 1. Problem definition • Clarity of the problem definition • Translation of problem definition in evaluation questions • Description (& definition) of evaluation criteria 2. Evaluation subject • Definition and demarcation of the evaluation object • Description of the policy context and institutional context of the subject evaluated

7. Example of Validity in the evaluation assessment form (continued) 3. Policy theory • Description of (reconstructed) intervention logic • Operationalisation of the measurement of results by means of indicators (distinction between different results levels) • Analysis • Specification of evaluation methods applied • Justification of the applied methods and techniques (including description of the limitations of the evaluation) • Data analysis, and interpretation & formulation of findings (= underpinning of the conclusions) • Consistency between conclusions and recommendations

8. Ex-ante and ex-post application of standards to assess evaluation quality • Ex-ante • Standards to be used by operational staff to prepare Evaluation Terms of Reference • Standards applied by IOB to advise operational staff on Evaluation Terms of Reference & evaluation process • Standards to be followed by evaluation teams • Ex-post • Standards applied by IOB to assess the quality of • Decentral (programme) evaluations at MoFA • Programme evaluations conducted by Dutch NGOs

9. 4. DAC Evaluation Quality Standards: some observations • Standards: a ‘normative framework’ giving guidance to realise ‘good practice’ evaluations • Provide pointers to develop and conduct evaluations • Are a useful background for staff which is not (very) familiar with evaluation and evaluation processes • Provide useful guidance for evaluation teams But…. • Not readily applicable as ex-ante instrument to realise quality evaluations or ex-post instrument to assess the quality of evaluation reports

10. DAC Evaluation Quality Standards: some observations (continued) • Why? • Some sub-headings/explanatory texts are/contain assessment criteria - others do not • Qualitative terms/concepts need definition (for those not very familiar with evaluation) • The list of criteria is very long for quality assessment which should focus on crucial elements • If the Standards are to be used to assess evaluation quality (including meta analysis), they should ideally contain a set of indicators and rating criteria

11. DAC Evaluation Quality Standards: some observations (continued) • Further operationalisation of the Standards as ex-ante guiding instrument and ex-post assessment tool needed? • Issue for discussion in break-out group(s) • Possible input: assessment checklist applied in MoFA Netherlands (handout)