Evaluation Planning and Management John Wooten ~ Lynn Keeys

1. WELCOME Evaluation Planning and Management John Wooten ~ Lynn Keeys June 2012

2. Session 8: Evaluation Team Debriefing and Report Evaluation Road Map : Evaluation Stage 5 Objectives: • Review some important considerations in debriefing evaluation teams • Review requirements of core sections of the evaluation report • Understand how to transition smoothly from team debriefings to the draft and final evaluation reports

3. Evaluation Team Debriefing and Report Some DOs and DON’Ts to remember… • Don’tengage in detail meddling • Don’t disallow team’s structuring inputs • Do consider iterative debriefings • Dounderstand how report drafting is unfolding • Dobe prepared to guide the team in developing an acceptable product

4. Evaluation Team Debriefing and Report Which comes first? Why? Draft evaluation report? Team debriefing?

5. Evaluation Team Debriefing and Report Which comes first? Why? Draft evaluation report? Team debriefing? Tip: Consider requesting an annotated report outline before the debriefing

6. Evaluation Team Debriefing Evaluation team debriefing • Finalize/activate team debriefing plan • Iterative debriefings, successively higher level • Representatively participatory/transparent • Evaluation team inputs on structure • ~One hour (20 min. presentation) • Organize around either : • Findings, conclusions, recommendations, lessons • Key evaluation questions • Project components • Agree on modifications for next debriefing

7. Evaluation Team Debriefing Evaluation team debriefing … FINAL • Provide team brief, written summaryof decision points, guidance, OR • Request annotated outlineof draft report (highlight decision points, guidance from debriefings) • Promptly ID in writingmisunderstandings, internal conflicts, differences of opinion; resolve as indicated • Preview all datato be submitted in agreed formats • Identify data-related issues in advance Tip: Avoid making assumptions

8. Evaluation Team Report All conspiring to answer…

9. Evaluation Team Report Draft evaluation report…

10. Evaluation Team Report Draft evaluation report…Iterative outline • Final evaluation SOW (illustrative outline) • Evaluation work plan (finalized/approved) • Cursory review field experience (final +) • Judicious use of annotated outline (R.e.a.l.l.y. FINAL) So when is final FINAL?

11. Evaluation Team Report Draft evaluation report…References • USAID’s Monitoring and Evaluation TIPS #17, “Constructing and Evaluation Report” (Table 1, p. 8 guided outline) • Un-numbered TIPS (same title) (Annotated report outline; Management Systems Int’l, 4/2006) • “Checklist for Assessing USAID Evaluation Reports” (Hard copies @ Tab 7)

12. Evaluation Team Report Organizing key elements of the evaluation report Recommendations Proposed actions for management Conclusions Interpretations and judgments based on the findings Findings Empirical facts collected during the evaluation Attribution? Counterfactual? Fig. 1, TIPS #17, p. 2 Artfully constructed, to convey/reflect team’s clarity, skill, professional judgments, experience and passion, contributing to decision-making processes that lie ahead

13. Evaluation Team Report Draft evaluation report…Organization • Organize around either : • Findings, conclusions, recommendations, lessons • Key evaluation questions • Project components • Otherwise • Share your own perspectives • Share good, bad, uglysamples/experiences • Understand key sectional differences…

14. Evaluation Team Report Draft evaluation report…Findings Tips on Findings? • Based on analysis: • Facts -- empirical evidence • Typically emerge from data analysis • Begin to answer key evaluation questions

15. Evaluation Team Report Draft evaluation report…Findings Tips • Bundle findings that answer a macro-question • Examine how management transformed inputs-to-outputs • Examine how management “positioned” outputs to influence (+,-) outcomes (causality) • Relate findings to project design (theory of change, results framework, logical framework, critical assumptions) • Include clear, hard-hitting summary of data analyses

16. Evaluation Team Report Draft evaluation report…Conclusions Tips on Conclusions? • Based on judgments: • Involve deductive judgments (meanings) • Emerge when applying reasoning to interpret evaluation findings

17. Evaluation Team Report Draft evaluation report…Conclusions Tips • Communicate importance, benefit, impact of empirical findings about project, context, problem, management framework • Demonstrate understanding of context • Summarize team’s views on whether/why a project succeeded or failed achieve various design aspects • Clearly answer “So what?”questions • Clearly/logically support empirical findings • Reflect skills, expertise of team members

18. Evaluation Team Report Draft evaluation report…Recommendations Tips on Recommendations? • Based on reflective experiences: • Answer, “So what do we do about it?” • Are judgments on what needs adjusting to enhance likely achievement of results in the current or future project or related programs

19. Evaluation Team Report Draft evaluation report…Recommendations Tips But how to write the report?!? • Flow clearly/logically from findings/conclusions • Anticipate, respond to pending decisions cited in SOW • Are actionable (i.e., feasible, cite responsible agency or authority, within their ‘manageable interests’)

20. Evaluation Team Report Draft evaluation report…Writing But how to write the report?!? > BONUSTIP < …by report outline? …by technical specialty? Consider how to avoid a report that reads like… …a committee product?!

21. Evaluation Team Report Draft evaluation report…Review • Triangulate review of draft report based on: • Final evaluation SOW • Written agreements from debriefings • Evaluation report checklist • Organize peer reviews (w/strict time limit) • Allow ample time for stakeholders’ review (cultural, language, availability constraints) • Push to resolve ALL dissenting perspectives, and feedback conflicts (…and if not resolved?)

22. Evaluation Team Report Draft evaluation report…Review • Preparetimely, written, contradictions-free feedback (the “corporate” perspective ~ USAID etal) • Follow-up w/confirmation phone call/meeting • Limit track changes to minor editorials only; provide major, substantivecomments separately • Re-confirm timeframe for receipt of final report • Reconfirm LOE/budget availability

23. Evaluation Team Report Final evaluation report…Review/Acceptance Congrats on this significant milestone~ • Review per evaluation checklist • Verify responsiveness to the “corporate” feedback • Negotiate required refinements • Review, confirm content/completeness of all quantitative data in agreed formats • Verify all SOW terms are fully satisfied • Issue acceptance letter (per A&A Office guidance)

24. Evaluation Planning and Management John Wooten ~ Lynn Keeys  • Thank you~ (jwootenjr@yahoo.com) (lynnkeeys50@yahoo.com)