1 / 39

Surveys in Humanitarian Emergencies

elsa
Download Presentation

Surveys in Humanitarian Emergencies

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

    1. 1 Surveys in Humanitarian Emergencies Muireann Brennan International Emergency and Refugee Health Branch Centers for Disease Control and Prevention Acknowledgements: Oleg Bilukha, MD, PhD Ask participants if anyone has participated in a survey to gain a sense of the depth of understanding and experience in the group? The main aim of this session is to allow participants - To know the basic good practice principles of a representative and reliable survey To be able to recognize a “good” survey To be able to interpret survey findings for programme survey Participants will not necessarily be able to know exactly how to do a survey after this session but will understand the basic principles of a reliable survey to enable them to review or critique a survey report or use the findings Muireann Brennan International Emergency and Refugee Health Branch Centers for Disease Control and Prevention Acknowledgements: Oleg Bilukha, MD, PhD Ask participants if anyone has participated in a survey to gain a sense of the depth of understanding and experience in the group? The main aim of this session is to allow participants - To know the basic good practice principles of a representative and reliable survey To be able to recognize a “good” survey To be able to interpret survey findings for programme survey Participants will not necessarily be able to know exactly how to do a survey after this session but will understand the basic principles of a reliable survey to enable them to review or critique a survey report or use the findings

    2. 2 Methods of Data Collection Go though slide and compare survey to assessment and surveillance. Ask, “what essential piece of information do you need for surveillance that you don’t need for a survey?” Answer, you need to have a reasonable idea of the denominator for surveillance, but the denominator is collected during a survey.Go though slide and compare survey to assessment and surveillance. Ask, “what essential piece of information do you need for surveillance that you don’t need for a survey?” Answer, you need to have a reasonable idea of the denominator for surveillance, but the denominator is collected during a survey.

    3. 3 Overview When to do a survey The need for a standardized tool and methodology Key activities in designing and implementing health and nutrition surveys Survey interpretation These are issues that will be discussed during this session. These are issues that will be discussed during this session.

    4. 4 Surveys in Emergencies Why would you consider doing a survey? When would you do a survey? PAUSE to discuss answers before moving on to answers in next few slidesPAUSE to discuss answers before moving on to answers in next few slides

    5. 5 Why Do a Survey in an Emergency Estimate prevalence (example acute malnutrition) Determine mortality Prioritize interventions Collect baseline data for program planning Evaluate program success Advocate for intervention

    6. 6 When To Do a Survey in an Emergency? After the initial emergency phase has passed After basic needs for survival been met When major population movement has stopped When you are in a position to take action After preparatory information has been collected When security allows These may not be absolute. For example, you may need to do a survey when major population movement is going on. Ask “has anyone been in this situation?” Ask “What might you do to deal with this issue? Answer, try to collect most up to date population data. Decide how migration in and out of households will be dealth with in the questionare.These may not be absolute. For example, you may need to do a survey when major population movement is going on. Ask “has anyone been in this situation?” Ask “What might you do to deal with this issue? Answer, try to collect most up to date population data. Decide how migration in and out of households will be dealth with in the questionare.

    7. 7 Difference Between Emergency and Development Surveys Emergency surveys Must be done with action in mind Must be implemented and analyzed quickly Hard to find trained/trainable personnel Difficult to come up with sampling frame Multi sectoral, multi partner (UNICEF, UNFP, etc.) Security Ask “Has anyone been faced with these issues in the field?”. If yes, “how did you deal with them?”. Some CDC examples: Liberia, few trained personnel, longer training period. Darfur, few trained personnel, difficult to supervise, fewer, but well supervised teams. Darfur, sampling frame difficult to determine, used WFP food distribution lists, MOH mass measles campaign population lists, ICRC distribution lists, NGO population lists. Ask “Has anyone been faced with these issues in the field?”. If yes, “how did you deal with them?”. Some CDC examples: Liberia, few trained personnel, longer training period. Darfur, few trained personnel, difficult to supervise, fewer, but well supervised teams. Darfur, sampling frame difficult to determine, used WFP food distribution lists, MOH mass measles campaign population lists, ICRC distribution lists, NGO population lists.

    8. 8 The Problem: Non-standardization of Methods This less true now. However, the perceived need for agreed approaches such as SMART, suggest that it is still an issue. Ask “has anyone faced confusion in the field about what the standard approach is in conducting a cluster survey?”This less true now. However, the perceived need for agreed approaches such as SMART, suggest that it is still an issue. Ask “has anyone faced confusion in the field about what the standard approach is in conducting a cluster survey?”

    9. 9 Survey design and implementation: key tasks Determine objectives of the survey Determine broad questions to be answered outcomes to be measured Estimate Budget needed and Identify key partners Define the sampling frame & sampling design Design the questionnaire, translate, and test Conduct training , plan the logistics, equipment, and survey team needed Data entry, analysis, and Interpretation of results Preparation and dissemination of results Take action It is important to develop a clear timeline for these activities. Allow wide margins for error. Always allow more time for training that you think you will need. In Liberia, CDC had to stop a survey after the first day and bring teams back for further training This is always an option.It is important to develop a clear timeline for these activities. Allow wide margins for error. Always allow more time for training that you think you will need. In Liberia, CDC had to stop a survey after the first day and bring teams back for further training This is always an option.

    10. 10 What Might Be Included in an Emergency Survey? Demographics Household Characteristics Child Health Child Nutrition Reproductive health Mortality In Darfur, a questionnaire was developed with input from all partners, reflecting the multi-disciplinary set of topics. Once final the questionnaires were translated into Arabic and then back translated. However, ONLY collect information that will be used. Ask “About how many questions have people used in one questionaire?” In Darfur, a questionnaire was developed with input from all partners, reflecting the multi-disciplinary set of topics. Once final the questionnaires were translated into Arabic and then back translated. However, ONLY collect information that will be used. Ask “About how many questions have people used in one questionaire?”

    11. 11 Defining The Sampling Frame What population do you want to extrapolate your results to? What population is or will be the target of the program? Is the population homogeneous? Are there geography and access issues? Security conditions? When determining the sample frame to be used, or the geographic/demographic scope of the survey these factors should be considered. When determining the sample frame to be used, or the geographic/demographic scope of the survey these factors should be considered.

    12. 12 Sampling Frame for the Darfur Survey Dark orange represents inaccessible areas, yellow is more accessible. What population to extrapolate to? Each of the three states seperatly. Program target? IDPs and residents targeted for WFP food distribution. Population homogenous? No, hence need for separate state estimates. Security issues? Major, therefore estra clusters selectes, WFP helicopters planned for In the end, 3 separate surveys were completed, 1 for each state.Dark orange represents inaccessible areas, yellow is more accessible. What population to extrapolate to? Each of the three states seperatly. Program target? IDPs and residents targeted for WFP food distribution. Population homogenous? No, hence need for separate state estimates. Security issues? Major, therefore estra clusters selectes, WFP helicopters planned for In the end, 3 separate surveys were completed, 1 for each state.

    13. 13 Sampling Frame for the Ethiopia Survey Population not homogeneous Highlands versus lowlands However, intervention targeted by Woreda Sometimes cannot reconcile some of these needs. Ask: “has anyone worked in Ethiopia?” If yes, what are the differences between highland and lowland populations? Answer, agriculturalist versus pastoralist/migratory, often in the same district/Woreda. Problem. The unit of intervention was the Woreda. The Woreda either got or did not get assistance. Solution? No satisfactory solution. In practice, surveys done by Woreda. Sometimes cannot reconcile some of these needs. Ask: “has anyone worked in Ethiopia?” If yes, what are the differences between highland and lowland populations? Answer, agriculturalist versus pastoralist/migratory, often in the same district/Woreda. Problem. The unit of intervention was the Woreda. The Woreda either got or did not get assistance. Solution? No satisfactory solution. In practice, surveys done by Woreda.

    14. 14 Decisions Concerning Sampling Design Probability sampling: Simple random sampling Systematic random sampling Cluster sampling Sampling Methodologies can be divided into two broad groups (1) probability and (2) non-probability. The main difference between the two is that probably sampling requires that a “randomness” and is statistically representative i.e. no one has control over the selection of individuals or households. In general, non-probability sampling is used for collection of qualitative information. In many situations, these methods may be more appropriate Ask Can non-probability sampling be “representative” ? No, not in a statistical sense. It can however, be useful information information which reflects the situation. In this session, we will focus on probability sampling. Sampling Methodologies can be divided into two broad groups (1) probability and (2) non-probability. The main difference between the two is that probably sampling requires that a “randomness” and is statistically representative i.e. no one has control over the selection of individuals or households. In general, non-probability sampling is used for collection of qualitative information. In many situations, these methods may be more appropriate Ask Can non-probability sampling be “representative” ? No, not in a statistical sense. It can however, be useful information information which reflects the situation. In this session, we will focus on probability sampling.

    15. 15 Each individual or sampling unit in the population has the same chance or probability of being selected The selection of one individual should be independent of the selection of another Representative Sampling

    16. 16 Simple Random Sampling Most basic type of sampling Selection of units independent and random Steps: Number each sampling unit Choose new random number for each selection Simple random sampling - sampled units (usually individuals or households) are selected at random from a sampling frame (complete list of all individuals or households in the population), Implies the existence of complete up-to-date list of individuals or households, such as a population register, mosque list, telephone directory. Not readily available in emergencies. However, longer term UNHCR camps will have household lists. Simple random sampling - sampled units (usually individuals or households) are selected at random from a sampling frame (complete list of all individuals or households in the population), Implies the existence of complete up-to-date list of individuals or households, such as a population register, mosque list, telephone directory. Not readily available in emergencies. However, longer term UNHCR camps will have household lists.

    17. 17 Systematic Random Sampling Similar to simple random sampling First sampling unit chosen randomly Systematic selection of subsequent sampling units Steps: Compute sampling interval (SI) (Number in population / Sample size) Select random start between 1 and SI Systematic sampling - sampled units are selected the population or at equal intervals (the sampling interval); every nth individual or unit is selected. Does not necessarily require a sampling frame. May be possible in organized camps. Often used in well laid out refugee camps. Used in small geographical areas. Need to be able to count individual individuals or households, either on a list or on the ground. Systematic sampling - sampled units are selected the population or at equal intervals (the sampling interval); every nth individual or unit is selected. Does not necessarily require a sampling frame. May be possible in organized camps. Often used in well laid out refugee camps. Used in small geographical areas. Need to be able to count individual individuals or households, either on a list or on the ground.

    18. 18 Simple and systematic random sampling

    19. 19 Calculating the sample size For A Random Sample or Systematic Sample This slide shows the calculation used to calculate a sample size. Go through slowly. Emphasize don’t need to remember this. Just underatnd what needs to be taken into account. What is the risk of error? The percent chance you are willing to be wrong. For most surveys, 5% error risk is acceptable. This means that if I do 100 surveys, 95% of the time my estimate will be within x and Y of the true value. How is the expected prevalence determined? Using secondary sources, informed judgment, experience, findings from previous surveys, etc. If there is absolutely no information, then an estimation of 50% can be used. Assuming a prevalence closer to 50% is safest because 50% requires the largest sample size. What level of precision is acceptable? The precision of the estimate depends on the sample size and the survey design. In general, if the prevalence of a condition is high, lower levels of precision can be tolerated. For example, a 95% confidence interval of +/- 10 percentage points (absolute precision) would seem reasonable for an estimated prevalence of stunting near 50%, but the same absolute precision would be unacceptable for a prevalence level of 10%. But the level of precision you need really depends on what you will use the results for. If you want to compare later surveys to a baseline, you may need good precision for the baseline. If you only want a general idea if malnutrition or a specific micronutrient deficiency is a problem, you may not need so much precision. This slide shows the calculation used to calculate a sample size. Go through slowly. Emphasize don’t need to remember this. Just underatnd what needs to be taken into account. What is the risk of error? The percent chance you are willing to be wrong. For most surveys, 5% error risk is acceptable. This means that if I do 100 surveys, 95% of the time my estimate will be within x and Y of the true value. How is the expected prevalence determined? Using secondary sources, informed judgment, experience, findings from previous surveys, etc. If there is absolutely no information, then an estimation of 50% can be used. Assuming a prevalence closer to 50% is safest because 50% requires the largest sample size. What level of precision is acceptable? The precision of the estimate depends on the sample size and the survey design. In general, if the prevalence of a condition is high, lower levels of precision can be tolerated. For example, a 95% confidence interval of +/- 10 percentage points (absolute precision) would seem reasonable for an estimated prevalence of stunting near 50%, but the same absolute precision would be unacceptable for a prevalence level of 10%. But the level of precision you need really depends on what you will use the results for. If you want to compare later surveys to a baseline, you may need good precision for the baseline. If you only want a general idea if malnutrition or a specific micronutrient deficiency is a problem, you may not need so much precision.

    20. 20 An example of calculating the sample size Emphasize this is for a simple random or systematic random sample only. Emphasize this is for a simple random or systematic random sample only.

    21. 21 Cluster sampling Cluster sampling – sampled units are selected in groups known as clusters. For example 30 clusters of 15 children. Convenient because: fewer sites visited sampling frame or list is not required To take account of the clustering effect the sample size must be substantially increased. This is the most commonly used method in emergency situations Cluster sampling – sampled units are selected in groups known as clusters. For example 30 clusters of 15 children. Convenient because: fewer sites visited sampling frame or list is not required To take account of the clustering effect the sample size must be substantially increased. This is the most commonly used method in emergency situations

    22. 22 Simple Random Sampling

    23. 23 Cluster sampling

    24. 24 Calculating the sample size in cluster surveys

    25. 25 Photo taken in Macedonia of Cegrane Camp for Kosovar refugees: a systematic sample was used here as the tents were in neat rows and could be counted to choose which tents to include in the sample. Photo taken in Macedonia of Cegrane Camp for Kosovar refugees: a systematic sample was used here as the tents were in neat rows and could be counted to choose which tents to include in the sample.

    26. 26 This is generally more reflective of the situations where we are required to do a survey. Widely dispersed population with no obvious pattern or organization. Thiis is southern SomaliaThis is generally more reflective of the situations where we are required to do a survey. Widely dispersed population with no obvious pattern or organization. Thiis is southern Somalia

    27. 27 These are questions that you should ask your self when reviewing a surveyThese are questions that you should ask your self when reviewing a survey

    28. 28 Sampling - Goal

    29. 29 Sampling – Sample size So how do we decide on the needed precision? One-time results for advocacy alone does not need much precision (?0.10 good enough) Results that you will need to compare against in the future need greater precision (?0.05 if program will have large impact) Results you will monitor frequently (e.g. year by year) need even greater precision (?0.02)

    30. 30 Adjustments to Sample Size Source of information about number of analysis units (e.g. adult males) per samping unit (household) Calculate from census statistics Use previous surveys to calculate Finally, need to add margin for non-response Look at previous surveys May differ by region of country May be higher for some measures (e.g. blood)

    31. 31 Training & data analysis Training needs to include: what data to collect, how to select households, data collection method, consent form, interview method Data entry: advise to begin data entry simultaneously as data collection Data analysis: select the analysis tool (Epi-info, SPSS), identify key personnel Ask, “why should we do data entry as we go?” Answer, this way we up mistakes early. This is especially important in which kind of survey? A nutrition survey, because you will see early on if teams are mesuring height and weight with sufficient quality. Ask, “has anyone had the experience of picking up an error (or not) early in a survey?”Ask, “why should we do data entry as we go?” Answer, this way we up mistakes early. This is especially important in which kind of survey? A nutrition survey, because you will see early on if teams are mesuring height and weight with sufficient quality. Ask, “has anyone had the experience of picking up an error (or not) early in a survey?”

    32. 32 A good survey report always reports the confidence intervals. Pause before next slide and ask “what other factors need to be taken into account for interpretation?”A good survey report always reports the confidence intervals. Pause before next slide and ask “what other factors need to be taken into account for interpretation?”

    33. 33 Interpretation: Other Factors Trends and change Confidence Intervals (CI) Seasonality Aggravating factors or risks Baseline or ‘normal’ prevalence Prevalence of other types of malnutrition e.g. chronic malnutrition Mortality levels Ask, “why are aggraviting factors important?” Answer, because the cutt offs for declaring various levels of emergency allow for aggrevating factorsAsk, “why are aggraviting factors important?” Answer, because the cutt offs for declaring various levels of emergency allow for aggrevating factors

    34. 34 This is what one would like to see in a trend. This is what one would like to see in a trend.

    35. 35 Assessing the Quality Of Survey Implementation Did the survey answer the question it was designed to? Were objectives clear measurable and measured? Was the sampling frame adequate? Questionnaire Clear? In local language? Translated, back translated? Sampling design Representative? Sample size adequate? We will not address in depth each of these steps. We will not address in depth each of these steps.

    36. 36 Assessing the Quality Of Survey Implementation Logistics, equipment, and survey team Equipment standardized? Training adequate? Personnel? What checks for data quality have been done? Do you agree with the interpretation of results? Have results been disseminated to all partners What action is planned?

    37. 37 These are the same questions, just in table formThese are the same questions, just in table form

    38. 38 Assessing The Quality of the Survey Ask, what do you think of the survey height and age distribution compared to the reference population?” Answer, it is very uneven and has a wide standard deviation” Ask, which measurement is probably responsible for this? Answer, age, is very difficult to determine in emergencies. Ask, what do you think of the survey height and age distribution compared to the reference population?” Answer, it is very uneven and has a wide standard deviation” Ask, which measurement is probably responsible for this? Answer, age, is very difficult to determine in emergencies.

    39. 39 Take Action Interpret and understand findings Review and revise program objectives Advocate for resources e.g. food pipeline or access to clean water Address underlying causes of poor health or nutrition Increase coverage of programs Vaccination Close emergency nutrition programs Use findings as part of wider country information system Use findings as baseline data

More Related