Evaluation Planning and Management John Wooten ~ Lynn Keeys

1. WELCOME Evaluation Planning and Management John Wooten ~ Lynn Keeys June 2012

2. Session 6: Evaluation Team Selection and Building Evaluation Road Map: Stage 3 Objectives: • Overview basic requirements for selecting a (external) evaluation team • Preview important evaluation team-building steps, concluding in an evaluation work plan

3. Evaluation Team Selection and Building …Sequenced and tightlyi-n-t-e-g-r-a-t-e-d

4. Select External Evaluation Team… • Organize/conduct proposal(s) review • Ensure integrity of competitive selection process • Conduct due diligence checks (reference, background) • Conduct interviews • Flag implications for SOW, work plan, team briefings, sites visits, logistical arrangements, etc. • Rank proposals and candidates (per SOW criteria) • Establish competitive range and proposals ranking • Draft ranking memo (id problems, special needs, concerns) • Follow A&A office selections, negotiations • Confer on budget and negotiation hiccups • What about team • augmentation members?

5. Start Date and Pre-arrival Assignments… >><< TENSION >><<

6. Initial Team Briefing… • How many? • Who participates? • Whose agenda? • How long? • “What if?”, “what if?”, what if?

7. Initial Team Briefing… Some tips… • Formal introductions (bio-data summaries, formal titles and areas of responsibility, contact information, business cards, emergency contacts, etc.) • Key personalities, issues and opportunities (internal and external) • Political/cultural sensitivities • Known “skeletons in closets!” (Internal and external; caution reporting unsubstantiated but important insights; hear-say) • Open issues (e.g., prior evaluations, audits)

8. Initial Team Briefing… Some tips… • Team augmentation plans and implementation approaches (NO surprises~) • Lines of communication (internal/external) • Pending evaluation-based decisions (carefully E-L-A-B-O-R-A-T-E !) • Likely best users of evaluation and why • Special report guidance (recommendations, attachments, “unattached” attachments)

9. Initial Team Briefing… Some tips… • Tentative evaluation results communications plan (disseminations, summaries, vignettes, lessons learned, etc.) • Backgroundmaterials(Id important gaps, review of files) • Interimdebriefings arrangements (frequency, participatory, venues, equipment, context) • Prior approval requirements (Id flexibility vs. prior approvals areas) • Plans for iterative debriefings (dates/time, internal, external, mixed, venues, equipment, participants)

10. Evaluation Team Planning Meetings… EPM tasks: • Respect /facilitate team’s need for privacy and relative independence • Plan joint teams sessions • Facilitate integration of augmentation team members Evaluation team tasks: • Familiarization activities (varying perspectives)* • Understand the context from various perspectives • Data review; data collection methods and instruments; and data analysis protocols • Logistics/administrative * • Evaluation work plan*

11. Evaluation Team Planning Meetings… • Update the logistical/administrative arrangements • Share EPM plan with evaluation team • Discuss and finalize based on Data Sources, Instruments and Analysis Requirements plan • Develop evaluation work plan • Agree on outline • Consider requesting an annotated outline (saves time and LOE!) • Ensure timely approval • Consider contingencies -- “Plan B” Initial team huddles completed~

12. Evaluation Team Planning Meetings…

13. Evaluation Team Planning Meetings…

14. Evaluation Team Planning Meetings… • …New data collection process • Define target population to be sampled (census) • Determine “sample size” based on the degree of confidence required in results • ‘Control’ for sampling bias and error (probability vs. non-probabilitysampling; randomness) • Select the sample • Decide data collection methods • Design/administer data collection instruments • Note/document, for inclusion in evaluation report known data limitations

15. Evaluation Team Planning Meetings… • …Some data collection “ility” factors • Accessibility. How accessible are the data we need given the project context? • Timely availability. Will the data we need be available in a timely manner? • Applicability. Are the proposed data collection methods applicable to the kind and likely sources of the data? • Affordability. Can we afford to collect the data given our budget constraints?

16. Evaluation Team Planning Meetings… • …Some data collection “ility” factors • Acceptability. Are the questions we need to pose in order to collect the data we need acceptable from the socio-cultural and management perspectives (cross-cultural sensitivity)? • Practicality. Are the data we need practical in the project environment? What are the best practical ways to collect this data? • Reliability. What do we need to do to ensure this data will be reliable once collected? • Utility. How useful will the data be in helping to address key evaluation questions?

17. Evaluation Team Planning Meetings… Data Sources, Instruments and Analysis Requirements

18. Session 7: Evaluation Fieldwork Evaluation Road Map Stage 4 Objective: • Understand the responsibilities and potentially valuable contributions of the EPM team during evaluation field work

19. Evaluation Implementation Remain mutually engaged WARNING: Mutual disengagement during field work can have unfortunate and potentially disastrousimpacts on the evaluation experience and products

20. Evaluation Implementation Substantive field tasks… • Allocation of field tasks (Loosely monitor) • Quality of team leadership/followership • Assignments per skills/strengths (future assignment potential) • Quality/adequacy of participatory inputs • Quality/adequacy of team augmentation members • Conflict management • Between evaluation team members • Between evaluation team and significant external parties • Between significant external parties, encumber the team • Actual/potential implications for evaluation, report, relationships, project implementation, de-briefing, follow-up

21. Evaluation Implementation Substantive field tasks… `` • On-going data collection and analysis • Preliminary indicative trends, significant insights • Work plan hiccups • Adequacy/adjustments to evaluation approach • Potential process/data limitations (remember the “ilities”) • Plan B potential; possible management implications • Worst case scenario (another study!) • Data presentation approaches • Effective, impactful presentation • Clearly labeled data tables and charts • Graphs, cross-tabs and trend lines • Anecdotes, success stories, good quotes, case studies • `

22. Evaluation Implementation …Administrative field tasks • Supervise-down; manage-up • Highlight brief key interlocutors (positive, negative, sensitive) • Manage on “need to know” basis • Manage approved LOE, budget • Monitor funds advances • Ensure timely invoice review/approval • Maintain field, home office communications • “Finger on budget pulse” (cost overruns, formal modifications)

23. Evaluation Implementation …Administrative field tasks • Engage in interim debriefings • Insights to inform, shape final debriefings (content and practice) • Logistical, admin arrangements for final debriefings • Proactive follow-up; noprocrastination • Promote effective data management • Organization, clean-up, back-up, protection (dirty data) • Quantitative data storage preps (new requirement)

24. Evaluation Planning and Management John Wooten ~ Lynn Keeys  • Thank you~ (jwootenjr@yahoo.com) (lynnkeeys50@yahoo.com)