The WSP Global Scaling Up Handwashing Behavior Project

1. The WSP Global Scaling Up Handwashing Behavior Project Monitoring and Evaluation Six month team meeting May 29, 2007

2. M&E Organization • Impact Evaluation • Objectives defined primarily by Gates • IE Team is responsible for technical design, But: • coordination/integration with county programs is essential • Overlaps with project monitoring are significant • Project Monitoring • Primarily a tool for project managers • Managers should have ultimate authority (Is this true? Or should it be consultative?) over design, resources, strategic design and data collection • Overlaps with IE should benefit both activities

3. Impact Evaluation Objective: Learn what works, why and how well: • What are impacts of HW-BC interventions? • Behavior • Environment • Health, social, and economic welfare • What are the antecedents to behavioral change? • Howdid the interventions affect outcomes? • What is the cost-effectiveness of these interventions? • Share lessons learned with countries, donors and other partners.

4. Project Monitoring/MIS • Project Monitoring system(s) serves the project managers. • It helps managers ensure that the project is functioning as it should. • If done well, it expands, contracts, and refocuses at the discretion of project manager.

5. IE: Outcomes of Interest • Intermediate • Sanitation coverage and open defecation • Reduced fecal contamination of houses, hands, food, water (? - not yet funded – country permission is not yet clear) • Final • Reduce diarrhea • Reduce parasite loads(?) • Reduce wasting, stunting and anemia(?) • Improve cognitive development(?) • Improve wages and productivity • Social welfare (security, female schooling, etc.)

6. IE: TS-SM Logical Chain of Evidence A compelling logical chain of evidence improves credibility. We hope to demonstrate: • A plausible behavioral change/ consumer demand model (closely linked to the Results Framework) through the BCF/model; • Microbiological evidence of infection transmission channels (not in our current RF); • Statistical links between intermediate and final outcomes (sanitation  diarrhea reductions);

7. IE: Data Requirements • Two-round household survey • Household demographics, economics • Household handwashing, sanitation and water • Intervention Process indicators (knowledge, etc.) • Measures of microbiological contamination • Health, nutrition and cognitive status • Employment, wages, productivity • School enrollment & attendance; sense of security • Monthly monitoring • Diarrhea among children <5 yrs. • Community Sanitation • Handwashing behavior • Where useful, institution-based disease/absentee reporting

8. IE: Use of Intermediate Outputs • The Results Framework should tell us whether the intervention worked as expected • Some program diagnostic information can be gathered, but most of these will be available only after the project. Usefulness to Gates is limited

9. IE: ImplementationImplications Baseline - the baseline survey needs to be in place prior to full-scale implementation. But can trial implementation be monitored before full-scale roll-out? Treatments & controls - choosing treatment and control groups requires: • Buy-in/agreement from field, Gov’t, stakeholders, implementing institution(s), other development partners • Detailed information on roll out, target population, unit of intervention (hh / community / village / district…), number of units to be covered, inhabitants per unit, geographic area, etc. Country-Specific IE Concept notes - the specific interventions determine the IE design – specific implementation details at country level are crucial to the design of the country-specific concept notes

10. Impact Evaluation: Where are we? Steps already taken • Organizational model: variations on a common theme • Experimental designs • Mass-media and EE generally not randomized • Direct Consumer Contact activities randomized • Country PIs and supporting teams • Peru – Paul is leading, w/Sebastian Galiani, Alex and others; • Senegal – no decision yet; • Tanzania – Sebastian Martinez will lead; supporting staff proposed; Initial visit scheduled; • Vietnam – Pascaline Dupas will lead.

11. IE: Where are we? (cont’d) Steps to be taken • Country PIs – Senegal’s needs to be proposed. Peru’s and Vietnam’s needs to be contracted; Tanzania done in-house • Nearly all country supporting teams must be identified; • Survey firms • Tanzania – really only one leading candidate; non-competitive contract may be needed • Peru – already discussing w/survey firms • Vietnam and Senegal – nothing known • Questionnaire development • July 9-10 meeting to prepare final x-country draft • Translation/reformatting July/August • Interviewer training August/September • Fieldwork starting mid-late September at earliest

12. IE: Implications for implementation • Baseline - the baseline needs to be in place previous to implementation in the treatment areas.Note that the initial/trial roll-out can (and probably should) be implemented in non treatment/control areas before the baseline survey. • IE Concept note - the intervention defines the IE design – implementation details at country level are crucial to design country-specific concept notes • Treatments & controls - choosing treatment and control groups requires: • Buy-in/agreement from field, Gov’t, stakeholders, implementing institution(s), other development partners • Detailed information on roll out, unit of intervention (hh / community / village / district…), number of units to be covered, habitants per unit, geographic area, etc

13. Objective of Project Monitoring/MIS • Determine whether inputs (from RF) are implemented as and when planned (according to workplan/milestones) • Identify problems early, when they occur • Facilitate problem diagnoses and effective solutions • How? Assess, in real time, whether inputs are achieving intended outputs • Do HHs know of interventions? • Have they heard behavior-change messages? • Do they intend to change their behavior? • Is it reasonable to expect that they will? • Allows for intervention improvements in real time

14. Inputs Routine Monitoring: can be 100% coverage (if contracted) Intermed. Outputs: Awareness/ Intent HH Health Outcomes Final Outputs: Behavior Activity Behavior monitoring? General Model of Project Monitoring Eval. & Review Verification/QA5% - 25%

15. MIS: data collection tools • Contract-based (Tanz; Vietnam) • Build routine reports into field activities(strike a balance between informative and non-disruptive reports) • Quality Assurance (QA) required (objective: unbiased routine reports) • Public sector institution-based (Peru schools, Vietnam MOH facilities) • Incomplete/biased reporting concerns and solutions • Independent Surveys • Most costly data collection, but needed for IE and some QA

16. MIS: Where are we (cont’d)? • Program monitoring data collection systems • Peru: School program + contract-based community DCCs – two systems needed • Tanzania – All contract-based? • Vietnam – at least two systems probably needed (one for contract-based work; one for government-based work) • Senegal? • Do we need a strategic plan for intensive early KAP monitoring? • All should establish QA Plans

17. MIS tools (cont) Other monitoring activities • QR / AR • Gates required; Frequency: quarterly and annually • Based on country-specific RFs / milestones → need to review / revise (if appropriate) for the 2nd QR • HAMR • Project required; Frequency: biweekly • Crucial for lessons learnt • Monitor Enabling Environment • Monitor Implementation Effectiveness

18. MIS tools (cont) Other monitoring tools • Budget Monitoring • WSP required; Frequency: currently every QR and AR • New TF structure under discussion: only one TF per country. Implications → simpler but calls for more rigor to not exceed (+/-) 10% limit imposed by Gates • Budget needs to be revised at country and global level before submission of 2nd QR (mid June)