Evaluation Planning and Management John Wooten ~ Lynn Keeys

1. WELCOME Evaluation Planning and Management John Wooten ~ Lynn Keeys June 2012

2. Session 9: Evaluation Follow-up Evaluation Road Map: Stage 6 Objective: • Review some priority evaluation follow-up requirements

3. Evaluation Follow-up EPM on Easy Street? Not just yet… “Strike while the evaluation trail is hot~”

4. Evaluation Follow-up Activate Evaluation Follow-up Plan… Disseminate results Prioritize recommendations Negotiate, formalize decision/implementation assignments Agree on administration framework/approach Identify/share key lessons Assess the evaluation report/experience Identify immediate project actions Store evaluation data * Develop/distribute success stories, vignettes Organize formal/informal appreciations * Peruse selective bibliography content…

5. Disseminate Results Activate Evaluation Follow-up Plan… • IPs and sub-IPs(1) • DEC/Washington(1) • All or key stakeholders (1) • Select beneficiary reps (2) • Select donors in sector (3) • Select NGOs (3) • Select private sector reps (2) • Select replication reps (1) • Mission website (4) • Project website (4) • Select missions in region (1) • Official files (1,2) • Communications officers (1) • US embassy counterpart (2) • Washington reps (1) • Key: 1 = Full report • 2 = Summary or excerpts • 3 = Summary noting full report available from Mission • 4 = Summary noting full report available from DEC

6. Prioritize Recommendations • Organize by: • Decision-making urgency • Likely costs • Potential stakeholder impacts • Potential beneficiary impacts • Mission internal/external focus • Political/cultural sensitivity

7. Negotiate Assignments • Assign by: NB: Ensure assignees have authority… • Agency • Section • Team • Person (w/supervisory ok)

8. Select Monitoring Framework • Monitor by: • Implementation schedule • Benchmarks • Means (staff/special meetings, memo/e-mail; frequency…)

9. Identify/Share Key Lessons • Share with: • Other donors with tangential (non-direct) interests • Potentially replication/extension groups • Local NGOs, NGO associations • Colleges/universities • Community-based organizations • Donor group meetings (plenary, sector) • Journalists (newsprint exposés) • Scholarly journals • USAID/Washington knowledge management systems Requires e-x-p-a-n-s-i-v-e thinking Get help~

10. Assess the Evaluation • Assess the: • Share selectively, with sensitivity and tact • Team, team members • Report • Overall experience • Contractor (A&A office assessment form) • Participatory approach experience • Evaluation team augmentation experience • Who needs to know; who would benefit?

11. Id Immediate Project Actions • Explore implications for… • NB: USAID auditors study evaluation reports~ • On-going and future project implementation (what) needs to and can change immediately?) • Reporting (reports content and scope) • Management and performance indicators, baselines, targets, interim targets (design structure) • Audit requirements (performance, financial, internal, external, concurrent)

12. Store Evaluation Data • Quantitative Data… NB: Washington guidance pending • Check for completeness • Back-up • Audit-proof • Upload to new central evaluation data warehouse

13. Develop Success Stories/Vignettes • Be creative… Format content for: • Local ~ Regional • US ~ Global • TV ~ Radio • Newsprint • Newsletters • Websites (Mission/Project) • Performance reports Use in: • Social events • Project ceremonies • Embassy events • Host country events • Community events

14. Express Appreciations • Make your list...check it thrice • ! • ~Recognize the value ~o~ Be culturally sensitive~ Evaluation planning/management team Evaluation team IP reps Host country reps Other donor / project reps Logistic facilitators USAID/Washington peer contributors Secretaries/administrative assistants Schedulers

15. Peruse the Bibliography • You, a continuing learner…licensed to learn Schedule time for continued learning Experts you hired conducted research Learn from what they learned NB: USAID is a ‘learning organization’

16. Implications for Performance Monitoring? Ponder: Now that you have a better understanding of planning and managing an evaluation, including using evaluation information… …how can this insight help you better perform your regular monitoring responsibilities?

17. Implications for Performance Monitoring? Consider: New strategy, evaluation and project design guidance targets program development and evaluation elements of the program cycle Comprehensive M&E contracts outsource in-house monitoring responsibilities, potentially leaving staff less interested in/less capable of performing basic monitoring functions Project officers are often too busy to perform quality monitoring, read performance reports and reflect on performance data until portfolio reviews Monitoring is often tightly focused on PMP performance indicators

18. Implications for Performance Monitoring? Possible Results: Site visits may become “funded escapes” from heavy office work loads and schedules Postponing all “Why?” questions until evaluation time may trigger important and expensive missed opportunities More evaluations may be tasked to “Tell me what I missed as a result of weak, ineffectual monitoring”

19. Implications for Performance Monitoring? RegularMonitoring “Evaluative Monitoring” Possible Response: Increase the “evaluative content” of monitoring

20. Implications for Performance Monitoring? Potential Benefits: Officers empowered/enabled to pose some “Why?” questions using locally available/affordable tools to help get timely answers Officers prompted to take more time to read and think reflectively on performance data Less potential need for expensive evaluations Evaluation resources focused on bigger, higher value “WHY?”questions Inherently governmental functions brought back into the Agency

21. Session 10: Common Pitfalls in Evaluation Planning and Management Objectives: • Conduct a content review by exploring common pitfalls • Identify plausible approaches, and commit to avoid identified pitfalls • Re-tune our thinking to be alert for other potential pitfalls

22. Common Pitfalls • Nothing new under the sun? …but there are a lot of old things we don't know to do~ Let’s consider some of your repeater problems… EPM pit

23. Common Pitfalls • Exercise… EPM Based on your experiences, suggest: • Why these common evaluation pitfalls occur? • What could result from falling in these pitfalls? • How you will avoid these pitfalls yourself? • How you could help others to avoid them? pit

24. Common Pitfalls • Exercise… EPM Identify pits this EPM fell into: • What went wrong? • Who dropped the ball? • How will you avoid these pits? pit

25. Session 11: Course Closure and Evaluation Objectives: • Overview reference materials • Conduct a post-test • Conduct course evaluation

26. Continuing Learning “In today's challenging and complex world, being highly effective is the price of entry to the playing field. To thrive, innovate, excel, and lead in this new reality, we must reach beyond effectiveness toward fulfillment, contribution, and greatness. Research is showing, however, that the majority of people are not thriving. They are neither fulfilled nor excited. Tapping into the higher reaches of human motivation requires a new mindset, a new skill-set…” Stephen Covey, Author

27. Continuing Learning • A few good habits… • Systematically continuepersonallearning • Peruse reference materials (Tabs 6 and 7) “A fire hose sip”… to whet your appetite; share with colleagues • Review evaluation report bibliographic materials • Browse recent evaluations (summaries; bibliographies) • Browse donor websites (relevant evaluations) • Seek/give coaching - mentoring (Give it away~)

28. Post-Test • Closed “book & mouth”, please  • Questions: • Multiple choice • Fill in the blanks • Cross references

29. Course Evaluation • Name and organization (Optional & appreciated~) • Rating per session (Rating scale 1 – 4) • Rating the: • Overall course • Course materials • Facilitation • Exercises • Venue • Breaks • Improvement opportunities

30. Evaluation Planning and Management John Wooten ~ Lynn Keeys  • Thank you~ (jwootenjr@yahoo.com) (lynnkeeys50@yahoo.com)