Overview of Questionnaire Design Issues

1. Overview of Questionnaire Design Issues • Content • Question Wording • Question Sequencing and Flow • Ease of Questionnaire Administration • Other Design Issues

2. Content Decide what to include and exclude to meet your survey objectives. • Analysts and data users should be consulted as early as possible in the process of specifying the subject matter. • All aspects of the survey subject must be investigated to ensure that robust information is gathered. (childcare). • You also need to be sure that only questions are asked which respondents are knowledgeable enough to answer (floor space in square meters). • Questions about past specific events may need to be accompanied by a memory trigger, such as a calendar of events. • Designers must carefully sequence sensitive questions.

3. Question Wording Questions must have the same meaning and provide comparable data from every unit in the sample. • Vocabulary used must be familiar to the average respondent, without regional or cultural variations in meaning. • The designer should take into consideration the variety of “frames of reference” existing among respondents (neighborhood). • Response categories offered, type (open or closed), number, and order. • Length of question - introduction or explanation, some respondents may feel this gives them time to think while others become inattentive or confused. • Different types of respondents may respond differently introducing bias.

4. Ease of Questionnaire Administration The questionnaire must be easy for the interviewer and respondent to use. • Placement of instructions • Reference to previous information, flipping back and forth • The harder it is for the interviewer to determine the flow of the interview, the more chances for introducing interviewer error, item nonresponse, and respondent frustration.

5. Other Design Issues Procedural decisions and questionnaire format are also important considerations. • Interview time may be limited due to cost or respondent burden, forcing reduction in questions asked. • The survey respondent should be considered, questions for each person may be worded differently than for those asked of a responsible respondent for a household. • The designer should also keep in mind the method of data collection, the unit of analysis, and how many times the sample unit will be interviewed. • Format can affect how well interviewers/respondents are able to follow instructions and answer questions.

6. Questionnaire Design Preparation Method of data collection • Self-enumeration • Interviews Method of data capture • Keyed data entry • Scanning (OMR and/or OCR)

7. What is a Questionnaire? • A questionnaire is a tool designed to transfer specific data from the respondent to the sponsor/user/analyst. • Questionnaires must be designed to yield information of the highest quality possible with special emphasis on relevance, timeliness and accuracy. • An effective questionnaire is one that is economical in human resources, produces data which is wanted and is processable.

8. Major Components of the Questionnaire: Introduction • Identify the survey and who is responsible for the survey. • Identify any laws that require participation in the survey and any assurances of confidentiality • Provide a statement of purpose which explains how the data are to be used and how it will benefit the respondent. • Provide contact information for questions and assistance

9. Major Components of the Questionnaire: Administrative data • Survey identification codes, geographical codes and a serial number for the questionnaire • Name of the respondent, address or location of the housing unit, establishment, etc. for which the data is being recorded • Certification of the enumerator and sometimes the enumerator’s supervisor/crew leader

10. Major Components of the Questionnaire: Main body • Contains substantive questions on the subjects of the questionnaire. • OPENING QUESTIONS are used to put the respondent at ease. These questions tend to be factual questions on demographic data applicable to all respondents. • CLASSIFICATION QUESTIONS are the questions that allow respondents to be grouped into different categories of interest based on their answers.

11. Types of Questionnaires: Verbatim: • The enumerator reads the questions off the questionnaire word for word. “What is …’s marital status?” If single ask: “Has … ever been married?” Schedule: • The questionnaire gives topics covered and response categories without specific question wording. Marital Status:

12. Method of Administration: Self-enumerated: • Clear and detailed information on how to complete the questionnaire is required • Appropriate writing instrument to use (pen/pencil) • Completion of responses (written, check-mark, oval) • Include detailed instructions on where to leave the questionnaire and alternative actions if unable to comply with the directions Interview: • Have the advantage of training enumerators • Still provide clear guidance on completion of the questionnaire but can use abbreviated instructions

13. Types of Questions Open-ended questions • produces an unstructured response that is recorded on the questionnaire in the words of the respondent Example: What have you done to look for work during the last 4 weeks? _____________________________

14. Types of Questions Open-ended questions are good: • for openers or change of subject • where not enough is known of the likely answers to form a response list • where responses are technical or complex • where respondent cooperation would be better than for a close-ended question • as a source of hypotheses and of wording for close-ended questions • to get more detail or elaboration of an answer • in avoiding leading respondents

15. Types of Questions Closed-ended questions • requires respondent to choose from a small number of choices, usually printed on the questionnaire Is … male or female? 1 Male 2 Female

16. Types of Questions Closed-ended questions are good because: • all respondents answer in the same frame of reference from the same choice of answers • variations in the respondent’s articulateness are irrelevant • the response categories are known before enumeration which simplifies data capture and analysis • it’s easier for interviewers and respondents to complete • it’s quicker and cheaper to administer • the sensitivity of some topics can be reduced • more complete information can be obtained through lists that can be read to the respondent

17. Types of Questions Filter or Screening questions • A two (or more) part question that is used to direct respondents to relevant questions. The answer to the filtering question determines which of two different questions a respondent next receives. Were you engaged in work for pay or profit the week preceding the survey? If yes, what was the name of your employer?

18. Skip Pattern The logical organization of an interview so questions are asked only of those who fit certain criteria. Other respondents are directed elsewhere in the questionnaire depending on the criteria they meet. • Did you work last week? (yes/no) • If yes, what type of work did you do last week? • If no, what was the reason you did not work last week?

19. Types of Questions Read-in question • Includes the answer categories as part of the question. In an interview situation, the response categories for these questions would be read to the respondent. At your job, are you: • Employer • Employee for pay • Self-employed or own account worker • Apprentice • Unpaid family worker

20. Types of Questions Sensitive questions • Questions which respondents may be reluctant to answer for various reasons: • Fear of consequences (legal or political action) • Fear that their honest response might not be socially acceptable • Question is considered too personal • Question is associated with painful memories

21. Translating Concepts into Questions • Applicable - Ask only for information that cannot be obtained elsewhere and are directly related to the objective of the survey. • Specific - Avoid questions which are open to interpretation or are too vague (sometimes, mostly, rarely). Always use the simplest terms possible to phrase the question. • Accurate - Do not ask for information or precision that respondent’s don’t know or are unwilling to provide. Avoid bias created by “wishful thinking”, questions on past behavior are more reliable than those on future behavior.

22. Translating Concepts into Questions • Precise - Ask only one question at a time, avoid conjunctions like “and” and “or”. Also avoid constructing negative questions as they lead to confusion. • Neutral - Avoid emotionally charged language which may offend respondents or cause them to answer in a certain way. • Unassuming - Questions should not assume that a certain condition exists, unless it has been determined in a previous question.

23. Methods of Recording Responses Type of question: • Closed-ended questions are pre-coded • Open-ended questions are write-in answers • Questions asking for numeric information are self-coded (age, income) • Some open-ended or administrative information may be coded by the interviewer in the field.

24. Methods of Recording Responses Data capture: • Manual coding required (timeliness, conversion error) • Method of data capture (keyed data entry or scanning) • Ease of keying, scanning in order to capture data accurately and quickly

25. Structure and Format Layout • Facilitate data collection, transformation and capture • Differentiate between questions, responses, sections and clarify skip patterns • Encourage better response rates

26. Structure and Format • Abbreviations and punctuation • Spell out abbreviations • Do not use commas or periods inappropriately • Abbreviated wording only with trained interviewers • Underline or bold critical words • Response categories • Margin-justify pre-coded and self-coded responses • Margin-justify code boxes for write-in responses • Space • Avoid cluttered appearance • Provide enough space for write-in responses • Avoid too many thick black lines - try thin gray lines

27. Structure and Format • Answer spaces • Should be clearly identifiable for each question • Use columnar or tabular format for answer spaces to avoid confusion • Ensure each response category box is clearly identifiable (dotted lines) • Interviewer working space • Provide space and instructions on the questionnaire for any calculations to be performed • Differentiation • Differentiate between the types of answers required and provide guides to skip patterns whenever possible.

28. Structure and Format • Color • Drop-out colors for image data capture • Differentiate between sections and clarify skip patterns • Colorful graphics can help motivate respondents to complete self-enumerated questionnaires • Question numbering • Each question should be numbered sequentially (1, 2, 3) • Each section should be numbered sequentially (I, II, III, A, B, C…)

29. Structure and Format • Instructions • Should be placed (in order of preference): • Above the question • At the beginning of a questionnaire section • On the front of the questionnaire • On a separate sheet or in an instructional booklet (referred to on the questionnaire) • Instructions should have a differentiated and standardized font, location or separation (box) • Language • Various languages may affect spacing and layout • Translations should be rigorously checked • Consistency • Formatting decisions must be consistent throughout.

30. Grouping and Ordering of Questions • Facilitates gathering information from respondents • Progresses logically from topic to topic • Only contains relevant questions • Interviewer can follow the questioning sequence

31. Grouping Questions • Divide questionnaire into sections, each dealing with a particular topic or subject. • Group together questions relating to the same time interval, reference period, or frame of reference. • Group questions by type of respondent required (proxy, individual) • Group questions by the unit of enumeration (households, house members, enterprises, holdings)

32. Ordering Questions • After grouping, question and questionnaire section order should be reviewed by designers and users. • Prioritize questionnaire sections and questions. • A questionnaire flowchart is useful for deciding question order and can be helpful during the development of edit specifications and data entry programs.

33. Questionnaire Flowchart Sample Q. 1-8a Q. 8b Yes No Q. 9-11a Q. 11b-c Yes No Q. 12-15a <5 yrs. old Q. 33 Yes & No, outside US Q. 16-18 Yes No, in US No Q. 15b Q. 19a Yes Q. 19b Yes Q. 19c No No Q. 20a

34. Burmese Questionnaire Flowchart Q. 1-8 Stop < 5 5+> Q. 9-11 Stop < 10 10+> Q. 12-16 Males & never married women Stop Ever married women Q. 17-19

35. Processing Considerations Keyed data entry • Appropriate number of code boxes per response • One box per character • Answer boxes margin justified • How many responses should be marked for each multiple choice question? • How will “not stated” be marked? • How will questions not applicable to a particular respondent be marked?

36. Processing Considerations Scanned data capture • Forms must be kept in good condition • If multiple pages, must be separated (bar codes, cutters) • Paper color, printing clarity, drop out colors • Sufficient white space around answer boxes • Proper writing instrument used to complete forms • More rigorous testing of questionnaires, early enough to make changes to software, equipment, and/or forms. • Compare against a keyed sample

37. Questionnaire Quality Issues with Scanned Data Capture Questionnaires still must be easy for interviewers and respondents to use but must also be very machine readable to maximize the advantages of speed and reduced human error with scanned data capture. Questions should be pre-coded as much as possible. • The major areas requiring quality control are: • Paper (color, weight and size) • Printing (color and clarity) • Format (page layout, response position(s), and clear zones)

38. Questionnaire Quality Issues with Scanned Data Capture Interviewers, and respondents for self-enumeration, must be trained, instructed in how to complete the form correctly. • OMR bubbles should be filled in completely with no stray marks, using the appropriate writing instrument. • OCR characters should be printed in upper case in individual character, spaced boxes. A standard should be developed for numbers and characters and alphanumeric fields should probably be avoided. Cursive alphanumeric fields can be recognized but this usually involves teaching the scanner to read local handwriting.

39. Approaches to Developing Questionnaires Some generally accepted rules exist that can guide the questionnaire designer in constructing an initial draft of a questionnaire. The development of any particular questionnaire is unique and refinement is necessary to ensure that a questionnaire used in the field will produce sufficiently accurate results.

40. Approaches to Developing Questionnaires There are a series of tools and tests which are useful in the initial drafting and subsequent refinement of a survey questionnaire. • The maximum effort needs to be dedicated to questionnaire development because once a questionnaire is in use, problems are costly or impossible to correct.

41. Tools for Developing Questionnaires • Unstructured Individual Interviewing • Qualitative Group Interviews • Participant Observation

42. Unstructured Individual Interviewing • A discussion of the proposed survey topics between individual members of the group to be surveyed and the questionnaire designer. • The discussion is guided by a topic outline rather than a set of specific questions. • This technique is primarily used to gain insights into the best way to structure the questionnaire. • Time required: 2 - 6 weeks, Cost: Salary and per diems

43. Qualitative Group Interviews • Informal discussions of selected topics between participants chosen from the population of interest and someone who is knowledgeable about group interviewing techniques and the purpose of the survey. • The information gathered can aid in developing the conceptual framework and data specification for a statistical survey and evaluating draft questionnaires. • Qualitative group interviews can also be used after a survey is completed to help analysts interpret the data. • Time: 2 - 4 months, Cost: Salary and per diems

44. Participant Observation Research • Anthropological technique for gathering information about persons with language, values, or experiences very different from the designer, or about whom very little is known. • Field researchers, knowledgeable in the local language, live with the persons of interest. They conduct key informant and unstructured interviews. • Information obtained can be used to ensure that the content of the questionnaire will provide enough information to satisfy the survey’s objectives and help phrase questions that can be understood by all respondents. • Time: 6 months - 1 year, Cost: Support of researcher and incidentals

45. Procedures for Testing the Questionnaire Draft • Informal Testing - subjective evaluations of the questionnaire, less control is necessary in choosing the sample and conducting interviews. • Formal Testing - relies on statistical evaluations, this report describes two variations: • Pilot Studies • Split sample tests

46. Informal Testing • A questionnaire field test involving a small number of interviews in the kind of setting chosen for the final survey (home, work, etc.). • The detection and correction of errors or weaknesses in the questionnaire depends mainly on subjective information provided by interviewers and observers. • This test is not designed to be evaluated on a rigorous statistical basis. • Usually done as a first test of the draft or as a final step to ensure that revisions work well.

47. Informal Testing - personnel • Requires a coordinator to select the test site, sample, select and train interviewers, structure a system to receive feedback and set up an evaluation plan. Experience or knowledge about data collection is an essential qualification. • Interviewers with varying experience and skill levels are desirable, however all interviewers should know how to probe to obtain information that will be used to refine the questionnaire. Observers may also need to be selected and trained. • Knowledgeable personnel are required to carry out the evaluation of the results. Skills include the ability to recognize problems in an interview, during review of completed questionnaires or tabulations and the implications of the test results for the questionnaire design.

48. Informal Testing - cost and time • Usually, adequate subjective information can be obtained from 50 to 300 respondents. • Respondents are generally selected purposively rather than randomly to achieve the desired objectives of the test. The sample may include either a broad range of sub-populations in the test or questions may be directed to a specific sub-population. • Informal tests can be used to evaluate the entire questionnaire or only a portion. More than one type of questionnaire may also be evaluated (split-sample testing). • Time: variable depending on particulars of the questionnaire and sample, however operational aspects may be around 4 months. Cost: also variable but the relative speed of the test keeps them low.

49. Formal Testing Field test which relies on probability sampling for respondent selection and for which results can be evaluated on a rigorous statistical basis. • Pilot study - a prototype of the survey conducted to observe all of the proposed survey operations working together. It calls for developing a design which duplicates the final proposed survey from beginning to end, including plans for data processing and analysis. • Split-Sample Test - conducted specifically to determine the “best” of two or more alternate versions of the questionnaire, or almost any aspect of survey operations.

50. Formal Testing - Pilot Study • Advantages of a pilot study include the reduction of time between the final survey and availability of results, since the post-collection procedures and analysis plan have been tested and refined. • Disadvantages are that there is often little time scheduled between the formulation of the survey plan and the data collection effort for a pilot test, evaluation and refinement of survey tools and procedures. A pilot study is also costly as it encompasses all survey procedures. • It is the large, complicated, or repetitive survey that warrants the pilot study investment, or situations where the efficient operation and flow of each phase is crucial to the success and cost of the operation