Managing Evaluations for Consistently High Quality

1. American Evaluation Association Annual Conference - 2013 Managing Evaluations for Consistently High Quality Molly Hageboeck

2. USAID M&E Projects Overseas Managed by MSI Pakistan Afghanistan Uganda Columbia South Sudan Kenya Ethiopia

3. USAID M&E Projects Overseas Managed by MSI • Keys to Success • Evaluations are projects – they can be managed • Identify key intervention points – quality checkpoints • Create tools for exerting quality control at • the checkpoints • Share the tools • with clients and • evaluation teams • -- Field handbook • -- New website • MSI build for • USAID E3 to • improve M&E includes evaluation management tools.

4. MSI Evaluation Management Checkpoints for USAID’s Process

5. Quality Checkpoint 1Evaluation Statement of Work (SOW) • Common Problems • Management purpose is • not clear/transparent • Evaluation Questions – • to many, not matched • to purpose, not • feasible • There isn’t always an • opportunity to comment • on or negotiate the SOW • Solution: Help your Clients • Improve the SOWS they • Prepare MSI Checklist for Developing/Reviewing Evaluation SOWS  Built it in about 2000  Gave it to USAID in 2010

6. Quality Checkpoint 2Written Review of Existing Information Before Final Design • Common Problems • Late receipt of project • reports/performance • data • Team reviews often • cursory – important • data not extracted & • shared • Solutions: • Ask for reports • when the SOW • is issued. • Develop/require • a structured desk • review product within • a short time frame MSI Desk Review Template – First Deliverable from Teams – Before Final Design

7. Quality Checkpoint 3Final Evaluation Design/Plan Prior to Field Work • Common Problems • The field team did not • prepare the proposal • stage design – and • may not follow it • Teams too often • start the field work • without a final design, • data collection and • analysis (and sampling • plan and all necessary • instruments • Solution: • Detailed evaluation design and • formal review/approval on a • question by question basis from • the actual team including all • instruments before they get the • keys to the jeep. • Provide teams with a structured format to get started

8. MSI “Getting to Answers” Matrix  Built it in about 2005  Gave it to USAID in 2010

9. Quality Checkpoint 4Post-Field Work and Analysis Pre-Draft Briefing • Common Problems • Teams start writing • before they work • out a clear flow • of findings, conclusions • and recommendations • grounded in their • evaluation evidence. • Many reports not • well supported by • evidence • Many mix up findings, • conclusions and • recommendations – • and confuse readers. • Solution: • Required oral briefing in bullets • to ensure all questions have • been addressed and F-C-R flow • Is logical • Block remaining LOE until this • step is passed – as the team may • need to get more data before • it writes.

10. Quality Checkpoint 5Structured Quality Focused Review of Draft Report • Common Problems • Clients tend to • review draft evaluation • reports on substantive • reports often skipping • over structural and • professional quality • aspects. • Quality fine points may • not get attention until • the final stage – when • all LOE has been spent • Or they remain missed • until a meta-evaluation • finds the flaws • Solution: • Evaluation quality review checklist – shared with teams the day they start and all members of draft report review teams. • Checklist based feedback to • team – and repeat use of • checklist with final report to • verify that improvements have • been made MSI Checklist for Reviewing Evaluation Reports  Built it in about 2000  Gave it to USAID in 2010

11. Current “News” on MSI’s Evaluation Management System • Update of MSI Handbook for Field Teams is underway • Recent meta-evaluation for USAID of 2009-2012 evaluations found problems • that greater internal use of an evaluation management system and associated • tools would have caught -- and a recommendation on strengthen internal • evaluation management practices in USAID has been provided.