Download
1 / 27
270 likes | 303 Views
GSF Surveys and Evaluation 11 th – 15 th September, 2017 Dar es Salaam, Tanzania. Session Objectives. To become familiarized with the new outcome survey proposed methodology, tools and GSF technical support system.
E N D
GSF Surveys and Evaluation11th – 15th September, 2017Dar es Salaam, Tanzania
Session Objectives To become familiarized with the new outcome survey proposed methodology, tools and GSF technical support system. To learn from the pilot outcome survey in Malawi as well as other recent GSF surveys (Nepal Sustainability Survey). To discuss experience of mid-term evaluations To learn more on the process of the recent Togo Joint Evaluation
Outcome Survey Objectives To provide statistically reliable data on key sanitation and hygiene household & facility level indicators to measure outcomes of GSF supported programmes in the GSF targeted areas. This also allows for the independent verification of reported program results; To serve as a sustainability check through identifying whether households in ODF declared communities have continued to use and properly maintain improved toilets and hand washing facilities and continue to comply to national ODF criteria;
Outcome Survey Objectives To understand if the results achieved have addressed equity and non-discrimination and have met the needs of marginal and vulnerable households/populations. To collect data on emerging areas or indicators that are too sensitive, challenging or not appropriate to collect through routine programme monitoring systems. For example indicators relating to satisfaction with sanitation and hygiene facilities and participation in programme interventions.
So not only measuring outcomes…. do ‘outcome surveys’ need a new name? Suggestions welcome
Methodology: Survey Tools Household head questionnaire Female/caregiver questionnaire Persons with disability/mobility issues questionnaire Schools and health facility questionnaire Structured observations
Household Head Questionnaire Household demographics and household assets Water access, treatment and storage Sanitation facilities (with inspections) Handwashing facilities (with inspections) Participation and exposure to programme activities Latrine construction, cost, repairs Sanitation habits and social norms
Female/Caregiver Questionnaire Latrine use of all household members Participation and exposure to program activities; Sanitation habits and social norms Satisfaction with sanitation and hygiene services in the home (privacy, safety etc.) Menstrual hygiene management Disposal of child feces Questionnaire to be administered by female interviewers only
Disability Questionnaire Access to sanitation and hygiene services in the home Satisfaction with sanitation and hygiene services in the home To be asked to persons (over 18 years) with a mobility limitation
Schools and health facility questionnaire Water management Sanitation management Handwashing Programme exposure MHM Questionnaires asked to school/health facility management
Structured Observations Conducted by a female observers who spend 3 hours in a household. Observers use a structured observation guide which is based on a checklist for recording and describing the following behaviors for all household members: Latrine use Handwashing with soap and water around events of potential pathogen transmission Safe disposal of feces of children under five years of age
Social Norms Example questions: Do you agree with the statement: Everyone should use a toilet Where do your neighbors think you should defecate How many of your neighbors would agree with the statement: It is fine for everyone to defecate in the open. If everyone in your community practiced open defecation, would you definitely practice open defecation To assess the existence of a norm, it is important to ask people not just what their personal normative beliefs are, but what they expect other's normative beliefs to be. Empirical norms: What people expect others in the same situation to do. Normative norms: What they think other people believe ought to be done.
Preliminary structured observation results, Malawi, 2017
Methodology: Sampling Key principles: Sample frame to include all communities/villages in which GSF-funded programs have been implemented (irrespective of ODF status); Random sampling of communities/villages using a probability proportional to size, multi-stage, cluster approach; Over sampling of ODF villages may be required so indicators on ODF sustainability can be reliably measured.
Methodology: Sampling Key principles of the design: Among communities/villages selected, 16 households at random will be selected from an updated household listing. For the structured household observations, a subset of 4 households will be randomly selected from the (16) interviewed households in each community. For the schools and health posts, if communities have more than one school or health post then sampling will be conducted so just one school or health post is selected per community
Methodology: Survey Management Recruit an independent research firm/agency Manage the research firm to do the following: Develop the survey plan Customize standard tools (and translate) Conduct a pre-test Conduct training Conduct field work (data entry in tablets) Analyze data and write the final report
Analytical Approach Latrine and handwashing access data analysed against JMP ladders Outcomes disaggregated against background variables (district, education of household head, wealth quintile, ethnicity) Outcomes disaggregated by year of ODF declaration
Analytical Approach Creation of composite indicators: • The extent of exposure to the GSF-supported CLTS intervention • The strength of ‘latrine use habit’ • The strength of social norms relating to latrine use • Satisfaction with sanitation facilities • Improved MHM practices • ODF status • Slippage
GSF Technical Support Process GSF to develop following: Survey Terms of Reference Standard survey tools for customization Analysis plan and report template Technical assistance system to review following survey documents: • Survey plan/sample design • Survey customised tools • Analysis and final report Technical assistance in country during survey design and training • Final report
Lessons from the Malawi Pilot For all actors (EA, GSF and University) to be involved in the review of research firm bids and survey proposal. Research firm to be on board for at least 4- 6 months. Tools too overloaded and analysis is too heavy .
Evaluation GSF commissioned 10 programme mid-term evaluations between December 2013 – January 2016 Evaluations all conducted by the same external organisation
Evaluation Lessons Evaluations were not all conducted at the mid-term and findings were often too late to feed into major decision processes regarding extensions and expansions; Findings did not necessarily uncover anything “new” but provided independent evidence to advocate for strengthening particular weaknesses in programmes at all levels. Evaluation management responses demanded attention to the above areas but monitoring of management responses has not been well systemised. General ownership of the whole evaluation process by different actors has been weak.
Summary Previous GSF M&E Guidelines stipulatedoutcomesurveys to beconducted at the mid-term and end-termperiod. Movingforward programmes willberequired to do outcomesurveys on a more regular basis. Everytwoyearsisproposed. Outcomesurveys are not alwayslinked to evaluationsalthough if an evaluationis to take place, an outcomesurveyshouldbeconducted in the 6 monthsprior GSF to provide a strongertechnical support role to outcome surveys. Evaluation schedule and criteriastill to bedetermined. Movingtowards a joint evaluationapproachwhere possible.
