Adapting to Real-World Constraints

1. Adapting to Real-World Constraints Managing Common Challenges in the Evaluation Process Social Impact

2. Overview • Background • GEM-3 Project • SI Evaluation • Selected Challenges • Number of Evaluation Questions • Contingency Planning • Reconciling conflicting data • Audience examples and Q&A Social Impact

3. USAID/Philippines Growth with Equity in Mindanao III Program (GEM-3) • 5-year (2008 to 2012), $99m program in the ARMM and other conflict-affected areas. • Largest and most diverse program (2/3 Mission funding). • Followed $22.3m, GEM-1 (1995-2002) and $82m GEM-2 (2002-2007) programs. • Despite 18 years, $203m+ USG funding, no external evaluation had been conducted. • GEM-3 had three overarching objectives: • Accelerate economic growth; • Ensure that as many people as possible participate in and benefit from that growth; and • Bring about and consolidate peace in Mindanao. • Seven components: (1) Infrastructure; (2) Workforce; (3) Governance; (4) Business growth; (5) Former combatant reintegration; (6) Communications and public relations; and (7) Logistical and perational support. Social Impact

4. Background on Evaluation • In June 2012, Social Impact (SI)contracted to conduct a final evaluation to examine successes, shortcomings, and lessons learned. • Mixed-methods design involving desk review; key informant and group interviews; household survey; a cost-efficiency/ERR; and site observation. • 11-member evaluation team and over six weeks of fieldwork in Manila and Mindanao. • Video used for evaluation trainings. Social Impact

5. Evaluation Questions: How many is too many? • Common challenge for both commissioners and evaluators. • Need sufficient breadth and depth. • Finite resources and time. • Balance what is possible and what can be actually be answered. • Ultimately, the questions chosen will dictate the design and direction of evaluation will follow. • Given high visibility and large number of stakeholders, 54 questions were included in SOW. Social Impact

6. Tips For Refining Questions • Aim for 3-7 primary questions. • Prioritize and rank. • Questions should be realistic and answerable. • Link to the evaluation purpose. • Choose questions which reflect real stakeholders’ needs and interests. • Data simulation exercise. • Don’t use questions which contain two or more questions in one. • Avoid simple yes/no questions. Social Impact

7. Contingency Planning: Hope for the best, Plan for the worst • “The best laid schemes of mice and men/Often go awry” ~ R. Burns • Challenges on GEM-3 Evaluation: • Weather. • Interviewees asleep. • Logistical challenges. • Site inaccessibility due to conflict. • Asking sensitive questions. Social Impact

8. Tips for Planning • Plan for known challenges (weather conditions, holidays, school closing, harvest, etc.). • Prepare contingencies. • Alternate sites. • Remote data collection. • Use of local evaluators. • Secondary sources. Social Impact

9. The Truth, The Whole Truth, and Nothing But…. • Challenges on GEM-3 Evaluation: • Different answers given to different evaluators. • Military escorts. • Presence of cameraman. • Pleasing the interviewers. Social Impact

10. Tips for Reconciling Conflict Data • “The truth is rarely pure and never simple.” ~ Oscar Wilde • Introduction and Confidentiality • Sequencing • Varied phrasing • Use of local evaluators • Triangulation • Randomized Response Technique Social Impact

11. What about your experiences? • What are some of your experiences/solutions? • What other common challenges have you found? Social Impact