Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation

1. Good Practice Planning Tips for GRPP Decision-Makers…at time of Evaluation November 14, 2009 Dale Hill Independent Evaluaton Group, World Bank

2. Key Messages • 1st evaluation likely to be easier & more successful if “enabling conditions”are in place early • If not in place, situational analysis essential before planning • Can then build in ways to compensate & progress on conditions • When assessing timing & readiness for evaluation, need to plan well to assure independence & needed expertise • More important for GRPPs – both needed at all stages • For independence, Governing Body makes most key decisions • Early planning also needs to consider audience, Stakeholder involvement and dissemination needs.

3. Special Features of GRPPs in Relation to Evaluation Planning • High expectations of evaluation – to guide development aid allocation decisions ($b) and yield broad lessons • Open-ended programs with multi-stakeholder governance; scope and design evolve over time • Governance & management structure raise issues of organizational independence & adequate expertise • Broad objectives & reach; often complex multi-level operations – global, regional, country, local • Management faces challenges of navigating complex decision-making to respond to Governing Body

4. Revisiting Enabling Conditions • “Intentionality” – Formal Commitment to M&E and to act on findings • Roles for M&E Defined in Advance • Formal M&E Policy and/or Principles • Results Framework • Agreed Objectives at Founding • Monitoring Framework/logical framework • Follow through – Design System + Baseline Data • Examples Beyond GEF at end of slides for takeaway

5. Good Practice (GP): Enabling Conditions • Early Provision for framework, baseline data collection and monitoring • Expert Panel set up w/Evaluation role early; Roles Defined • UNAIDS (several evaluations completed) • Millennium Villages (early evaluation completed only) • GFATM (1st program-wide evaluation completed) • GAVI (1st evaluation completed; 2nd underway) Conditions Set up for First Evaluation (Rare Cases)

6. 1st Step: Situational Analysis • Because of high expectations, planning doubly important • Before or soon after call for evaluation, Mgmt. should do situational analysis & place evaluation on GB agenda Management should ask: • What is driving demand for evaluation at this time? • How should we take account of or build on past evaluations (if applicable)? • What is our readiness for an evaluation at this time? • If not ready, what options do we have?

7. Timing and Past Evaluations ….High Expectations, & Multiple Stakeholders • What is Driving Timing? Who asked for evaluation? • Single donor may want for accountability – w/deadlines set • But 1st best is evaluation commissioned by program – with FULL Governing Body consensus on need & timing • How to take account of past evaluations? • Take account of whether the past evaluations met all expectations – may want better quality, coverage this time • Δ in global context may dictate different approach, coverage • Multiple single-donor evals of GRPPs common – avoid duplication and consider program-level meta-evaluation

8. Readiness Assessment • Status of consensus on need, timing & approach • Objectives of program and log frame up to date? • New constituencies to involve and include? • Status/Access to funding • 1st best – program budget for evaluation–but if not pre-planned – • Can get GB approval of special item – build in time for this • 2nd best – if donor(s) offer to fund, prevent undue influence • Status of GB & staff expertise on evaluation? Access to support? • Status of info: (Usual) Baseline/monitoring, program records • Sometimes overlooked – enough activities completed? • Consider audiences, translation, dissemination needs

9. Consequences of Poor Planning Poor Planning: w/no compensation for Constraints Consequences Low yield; limited findings Poor quality; compromised credibility; less used, accessible Eval. Q’s less clear; new data collection $$; findings limited Costly to program; burden on informants; may confuse strategy Constraints • Not enough activities completed • Inadequate budget for evaluation, dissemination • Monitoring system not in place • Not taking into account previous evaluations

10. Setting Broad Purpose & Design • Should flow from Situational Analysis • e.g. Accountability for donor; Assess Impact of Δ’d Context • Ideally Approved by Governing Body • Usually varies by Program Stage or Maturity

11. Assigning Roles: Key Players in the Evaluation • Governing Body • Overseers of Evaluation • Oversight Committee • Standing Technical Expert Panel – predefined roles or not • Program manager and staff • Independent Evaluation Team (usually consultants) • Others: Keep in mind for consultation, dissemination • Key partners not represented on GB (donors) • Other Key informants (e.g. past GB members…) • Implementing Agencies (need translation?) • Direct Beneficiaries (need different, briefer product?)

12. Preferred Roles of GB & Management

13. Roles of Others: Participation & Consultation Upstream & Downstream • Principles/mandate in charter, M&E Policy or legal agreements? • If not, Mgmt should consult GB and determine expectations & Consider: • Before (design), During (interviews/surveys), After (review/inform findings) • If no Formal Stakeholder List or Map – Prepare, to equip evaluators

14. Planning: Design, Budget, Contracting ...Expertise Needed on Evaluation ...Weighing all Options and Responding to GB imperatives • Design must respond to purpose & match complexity of program plus budget—credibility essential • If expertise available (Oversight committee, host agency) – can have confidence in design, budget adequacy and feasibility of work program • TOR can be specific; team selection mainly on qualifications • If not, several options: • 2-stage evaluation: “evaluability study” or experts draft TOR; • Leave details to evaluators & selection process • RFP has proponents submit design/work program (sometimes budget) • Inception report/early consultation on methodology desirable

15. Conclusions & Hopes for the Future • Ideally, the good practices we highlight will lead to more frequent & earlier attention to the enabling conditions which set the stage for successful GRPP evaluations. • Where that is not possible, we hope decision-makers will see the benefits of doing a situational analysis to inform early planning and ensure a realistic, consensus approach. • We further hope that such early planning will increasingly: • Lead to assigned roles in evaluation which ensure organizational independence & access to expertise at key stages; • Adequately address questions of participation and dissemination; • And thus lead to higher quality evaluations & better development.

16. Thank You