Questionnaire Design Part I

1. Questionnaire Design Part I Disclaimer: The questions shown in this section are not necessarily good or appropriate for a labour force survey. Some are shown as examples of what not to do. Do not take questions shown in this sections as recommendations for a survey.

2. Design Phase Define goals or purpose of survey • What kind of information do you want available for analysis? • What kind of analyses do you want to do? Evaluate resources and barriers • Agency capabilities and experience • Highly trained interviewers? • Experienced data entry personnel • Telephones? cell phones? addresses? • Funding • Language, fluency, and literacy • Hard to reach areas or subpopulations • Urban/rural or other geographic differences • Internal displacement, refugees, seasonal migration

3. Design Phase Survey format • Common modes - in person, telephone, self-administered • May be paper based or computer/handheld assisted • Make sure the format is suitable for the literacy level and technological access of respondents • Labor force survey questions will almost always be about attributes (what one is) and behavior (what one does) rather than attitudes (what one wants or prefers) or beliefs (what one thinks is true)

4. Design Phase Technology can make up for human resources lacks and save time and materials • Using computers or handheld devices for interviewing: • Removes the need for interviewers to follow skip patterns. This may be very important if interviewers do not have a lot of experience. • Allows for much more complex skip patterns. • Eliminates some data entry errors and legibility issues. • Makes record keeping easier. • Reduces the amount of paper interviewers must carry and keep track of. • Eliminates the need for the data to be hand entered into a computer later, reducing staffing needs, leaving more time to focus on data cleaning, preparation, and analysis, and reducing time from field to publication.

5. Design Phase Use one survey for everyone • There should not be separate surveys for urban and rural areas • For comparison reasons • Areas may shift from rural to urban or be in an in between phase. • There should not be separate surveys for different regions or ethnic groups • The survey should be designed to fit and take note of any special groups such as refugees, internally displaced people, and households comprised of orphans

6. Design Phase Language • Would segments of the population have difficulty understanding, responding to, or providing accurate information for the survey in English? • The most accurate results can be obtained when respondents take the survey in a language in which they are fluent. • Survey translation is time consuming, difficult, and expensive. • Best practice is to have several people independently translate from English to the target language. They then discuss the differences in their translations and agree on a common version that will work for all groups. • It is more important that the same sense of the questions and concepts is communicated than a literal word for word translation.

7. Design Phase Language • If there are many languages in use, it might not be possible to translate the survey into all of them. • Testing is required on the translation. • Verification the translation is accurate • Cognitive testing should be done • Interviewers must be able to interview in the required languages.

8. Define the Concepts Internationally accepted concepts • Well defined • Allow for comparability • Questions exist to measure them Country specific considerations • Are there topics where internationally accepted concepts may need to be modified? • Household structures may vary for reasons such as polygamy, orphans, and age at first marriage. • May additional questions be needed to cover a topic? • What employment arrangements are likely? • Is subsistence farming common?

9. Define the Concepts Country specific concepts (continued) • Countries can define their own concepts tailored to the realities of their economy and the needs of their government and private data users • This can be done while also maintaining the internationally accepted concepts. • Questions must be written and placed carefully so they do not interfere with others. • Concepts must be defined clearly so the effectiveness of the questions and questionnaire can be evaluated.

10. Sourcing Questions Use existing surveys and questions when possible • If a survey is appropriate, use it whole. • If changes must be made, all possible effects on the questions, concepts, and skip pattern must be thought out. • Mixing and matching questions between surveys is not a good idea. • Modular design allows sections to be added, removed, or replaced while minimizing risk • Skips are predominantly internal to the module so they are less likely to be broken. • Most of the impact of the question changes will be internal to the section.

11. Constructing Questions • Conceptualization Process of defining, formulating, and clarifying concepts by specifying what exactly should be measured and how • Reliability Repeated measurements should yield the same results? • Validity • What was supposed to be measured has to match with what is actually measured. • Is measurement bias – systematic error - low?

12. Constructing Questions: Example Example: Health care use • What reasons for use should be included? Only major or all? • What qualifies as a health care provider? Hospital? clinic ? doctor ? traditional healer ? nurse ? family member ? • Self only or all family members? What if parent takes child? • Measured by time? Visits? Illnesses? Treatments? • What is the reference period?

13. Constructing Questions • Use simple vocabulary • Don’t try to fit too much in a single question • Group questions on similar topics together • Provide “don’t know” and “refused” answer categories when they will likely be needed. • Use lead-ins to provide instructions, explain terms, and make segues between different sets of questions.

14. Question-Answer Process • Comprehension • Understanding words and phrases • What the respondent thinks you want • Retrieval of information • Did the respondent ever store the information in his mind? • Salience – the type of thing that a person remembers • How long since the occurrence? • Judgment • Generate internal answer, then assess it • Response Formation • Edit (possibly censor) • Communicate

15. Example: How frequent is your tea consumption? • Comprehension • Unusual/difficult wording - “How often do you drink tea?” is more common • Presupposition – assumes the person drinks tea • Retrieval • Might do it so often they don’t really think about it or pay attention • Might have different patterns on work days and non-work days and not think of this. • How long is the reference period? Recall. Guessing. Computational error.

16. Example: How frequent is your tea consumption? • Judgment • Supplement memories with guesses • Response • Formulate a response • Edit for interviewer. Might report more or less if they think their answer is strange.