Applied Opinion Research Training Workshop

1. Applied Opinion Research Training Workshop Day 2

2. Designing Quantitative Instruments Mary McIntosh

3. Quantitative Instruments • Designing a Questionnaire – Keep in Mind: • Don’t forget the big picture Follow your objectives and hypotheses to determine the content of your questions • The quality of your research depends on the quality of your questionnaire • Ask others to review and proofread your survey • Always pretest, do a trial run • Utilize other sources to assist you in designing your questionnaire Previous studies, expert input, etc. • Make instructions as clear as possible

4. Quantitative Instruments • Designing a Questionnaire – Step by Step: • Screening respondents • Wording questions • Response options • Order of questions • Length of questionnaire • How the questionnaire will be administered

5. Quantitative Instruments • Screening Respondents: Sample Parameters • First, you want to decide if you will need to specify certain criteria that your respondents need to satisfy • Age (e.g., over 18, under 65) • Profession (e.g., government official, private sector, etc.) • Status in household (e.g., mother) • These questions should come first

6. Quantitative Instruments • Wording Questions • Second, start writing your questions, keeping in mind: • Questions should be kept short • Use wording appropriate to your respondents’ level of education • Make all definitions, qualifiers, and assumptions explicit(e.g., Do you think the Bank’s program was effective? Which program?)

7. Quantitative Instruments • Wording Questions • Be careful when using abbreviations, acronyms, and jargon • Be careful with word choice • Highlight words that require extra emphasis • Make sure your questions are technically accurate • Make sure questions are relevant and applicable to your respondents

8. Quantitative Instruments • Wording Questions • The time span for recall in a question should reflect the saliency of the topic • Also be careful about asking questions about future intentions; keep to the short-term future • Time spans, past or future, should be as specific as possible (e.g., don’t use “lately”)

9. Quantitative Instruments • Wording Questions • Do not use leading questions that suggest a particular answer or the researcher’s viewpoint • (e.g., Why does working in the public sector make people lazy and careless?) • Do not use loaded questions that bias people towards particular answers • (e.g., Do you think it is important to have a strong police force in this time of crisis?) • Be wary of double-barreled questions. Rephrase them into two questions whenever possible • (e.g., Should the government reduce its financial help and its technical assistance to other countries?)

10. Quantitative Instruments • Wording Questions • Avoid double negatives (e.g., not prohibit), they can be confusing. If absolutely necessary, emphasize the “not” by underlining it • Make sure your questions are valid. The question must ask for information which the respondent is capable of providing an informed response • Make the question as specific as possible (e.g., instead of asking “Is privatization a good idea?” ask “Is privatization of utilities a good idea?”)

11. Quantitative Instruments • Wording Questions • Explicitly state alternatives • (e.g., instead of asking “To balance the budget, would you be in favor of a higher income tax?” ask “To balance the budget, would you be in favor of a higher income tax, a higher sales tax, or reduced government services?”) • Use face-saving phrasing • (e.g., instead of “Did you vote?” ask “Were you able to vote?”) • Be sure the wording of your question matches your response options • Consider “Don’t Know” options

12. Quantitative Instruments • Response Options Common response options are: • Likert/Rating scale • Dichotomous choice/Trade-off • Multiple choice • Open-ended • Ranking

13. Quantitative Instruments • Likert/Rating Scale • Shouldn’t use more than 2-3 scales in each survey effort Numbered Scales • Asks for responses on a numbered scale with descriptions for what the numbers represent (even and odd numbered scales) • e.g., Please indicate how favorable your impression of the World Bank is on a scale of 1-10, 1 being very unfavorable, 10 being very favorable • Numbered scales work well in written surveys • Numbered scales work well with educated population • Numbered scales must be appropriate for culture (e.g., 10 point scale vs. 7 or 8 point scale)

14. Quantitative Instruments • Likert/Rating Scale Word Scales • Word scalesare similar but meet different respondent needs. • e.g., Please indicate how much you agree with this statement, using a scale of “strongly disagree”, “somewhat disagree”, “neither agree nor disagree”, “somewhat agree”, “strongly agree” • What is your opinion of the Bank’s work in the area of infrastructure development in your country. Is it “very good,”“somewhat good,”“somewhat bad,” or “very bad.” • Word scales are usually 3, 4, or 5 point scales • Very common, easily used in written form or verbally • Word scales sometimes preferred because the opinion is clearly defined for respondent

15. Quantitative Instruments • Likert/Rating Scale • Provides a great deal of information, quickly and easily • Can be used to compute summary statistics (e.g., frequencies, means) and significance tests • Best used to measure opinions, attitudes, extent of agreement/intention, etc. Not as good for measuring facts • Consider a “Don’t Know” or “Refuse” response

16. Quantitative Instruments • Likert/Rating Scale • Common formats are 4-points, 5-points, 7-points, and 10-points (will vary based on how much knowledge respondents have; how nuanced their opinions may be. When in doubt, keep it simple (4 point scale.) • The greater the number of points, the greater variation you will be able to measure in respondents’ responses • An odd number of points allows respondents to give a middle or “neutral” response. Even numbered scale forces respondents to commit. • Odd point middle point can have a number of meanings: don’t know; neutral; neither positive nor negative

17. Quantitative Instruments • Likert/Rating Scale • Respondents need to be comfortable with the scales you utilize • If possible, use a balanced scale • e.g., 1-not at all effective, 5-very effective • If it is not possible to use a balanced scale, make sure your endpoints are clear antonyms • In your number descriptions, 1 should describe the smallest, most negative response and 4/5/7/10 should describe the largest, most positive response

18. Quantitative Instruments • Dichotomous Choice/Trade-Off • Asks for an either/or response • e.g., Were you able to vote in the last election? Yes/No • e.g., Which of the following two statements best represents your views on taxation in our country? 1) I believe our tax dollars are not used wisely by our government. 2) I believe our tax dollars are used appropriately by our government. • Flexible and easy to administer in written or verbal format

19. Quantitative Instruments • Dichotomous Choice/Trade-Off • Provides limited information - Does not tell you the extent of opinion • Yields dichotomous data • Can be used to compare response percentages and to examine how different groups responded • Best used to measure facts. Not good for measuring opinions, attitudes

20. Quantitative Instruments • Multiple Choice • Gives respondent a variety of responses to choose from • e.g., Which organization do you trust the most? (Oxfam, Save The Children, Catholic Charities) • e.g., What do you consider the most critical development challenge facing your country? (Poverty, Education, Corruption, etc.) • e.g., Which of the following describes your position? (Government official, Private sector, NGO, etc.)

21. Quantitative Instruments • Multiple Choice • Very common, flexible and easily used in written form; verbal ease depends on the number of choices • Provides a great deal of information, quickly and easily • Yields categorical or ordered data • Can be used to compare response percentages and to examine how different groups responded

22. Quantitative Instruments • Multiple Choice • Under some circumstances, you may want the respondent to be able to choose more than one answer • e.g., In your opinion, which are the most important Bank programs in your country? (choose up to 3) • Response options must be mutually exclusive • e.g., How much of your country’s GDP do you think is spent on foreign assistance: < 1% 1-4% 5-8% 9+%

23. Quantitative Instruments • Multiple Choice • Response options need to be exhaustive, but not overwhelming • Include an “Other” choice in which respondents can write in a response that you did not include • When applicable, order choices from smallest to largest, most negative to positive • “Other” and “Don’t Know” options should be last

24. Quantitative Instruments • Open-Ended • Allows respondent to provide any response that they want • e.g., Why do you think economic reforms have not been implemented by the government? • e.g., How would you improve the effectiveness of Bank programs? • Common, however, must be transcribed and, if not in English, translated • Limited to skill of interviewer and knowledge of respondent

25. Quantitative Instruments • Open-Ended • Provides a great deal of information • Yields data unique to each respondent; once it is coded, response percentages can be computed and compared • Risk: coding is subjective, time consuming, and expensive • Best used for exploratory research • Use when many answers are possible and multiple-choice options are too extensive or unknown

26. Quantitative Instruments • Ranking • Asks respondent to rank a variety of responses according to some criterion • e.g., Please rank the following in order of priority from 1-10, 1 being top priority: Education, Communicable Disease, Weak Institutions, The Economy, Corruption, etc. • Less common, best used in written form, difficult to administer verbally

27. Quantitative Instruments • Ranking • Can be difficult for respondents to complete • Provides limited information; assumes that respondents feel differently about each item • Data is difficult to analyze and interpret • Best used to measure limited attitudes and opinions

28. Quantitative Instruments • Order of Questions • If necessary, screening questions should be first to eliminate ineligible respondents early • Then, using the funnel approach, proceed from broad, general questions to more specific questions • Opening questions should be easy and non-threatening to keep respondents’ interest • If possible, make the opening questions interesting to peak respondents’ interest

29. Quantitative Instruments • Order of Questions • Consider the advantages/disadvantages of starting a questionnaire with an open-ended question • Avoid questions that get monotonous • Consider the advantages/disadvantages of completing all questions on an issue before moving to a new issue • Order issues in a logical fashion to facilitate the flow of the questionnaire • Obtain historical information in chronological order, either forward or backward

30. Quantitative Instruments • Order of Questions • Use transition statements to facilitate shifts from one issue to another or one series of questions to another • Eliminate order bias due to the sequencing of questions as much as possible • e.g., do not ask about economic growth as a major development challenge before you ask respondents what they consider to be the major development challenges for their country

31. Quantitative Instruments • Order of Questions • Do not ask sensitive or difficult questions until a rapport has been established with the respondent • Any demographic/classification questions should be asked at the end of the questionnaire, unless required for screening or cultural tradition • Demographic/classification questions should be ordered from the least to the most sensitive items • Avoid complex or frustrating skip patterns (if self-administered or face-to-face)

32. Quantitative Instruments • Length of Questionnaire • Don’t burden respondents with a long questionnaire • The shorter the questionnaire, the higher the quality of the data • Only ask questions that meet your specific research objectives • For each question, ask is this question necessary? • Pre-test your questionnaire, even if it is just among your colleagues, to determine how long it will take respondents to complete it • Language expansion factor • Cultural traditions of long introduction

33. Quantitative Instruments • Administering the Questionnaire • Confidentiality must be ensured • Consider not using a political polling firm • Keep in mind how you will be administering the questionnaire (e.g., mail-in, phone, in person, within groups, electronic) when deciding on the wording and length of the questionnaire

34. Quantitative Instruments • Administering the Questionnaire • Mail-in Questionnaires: • Best for longer questionnaires that may include sensitive issues for which respondents will want to maintain confidentiality • Should include postage-paid return envelopes to encourage participation • Phone Surveys: • Best for moderate length questionnaires, around twenty minutes • Should be done at appropriate times with interviewers fluent in the respondents’ language

35. Quantitative Instruments • Administering the Questionnaire • In-person Surveys: • Best for longer questionnaires • Should be done at appropriate times with local interviewers who are fluent in the respondents’ language • Skill of the interviewer will influence the quality of the data

36. Hands-on-Work:Designing a Quantitative Questionnaire

37. Designing Qualitative Instruments Sharon Felzer

38. Qualitative Instruments • Designing a Guideline – Keep in Mind: • This is a questionnaire for discussion • Don’t forget the big picture • Follow your objectives and hypotheses to determine the content of your questions • The quality of your research depends on the quality of your guideline and how well the guideline is administered • Ask others to review your guideline • Utilize other sources to assist you in designing your guideline • Previous studies, expert input, etc.

39. Qualitative Instruments • Designing a Guideline – Step by Step: • Introduction • Wording questions • Order of questions • Length of guideline • Administration of guideline

40. Qualitative Instruments • Introduction • The point of the introduction period is to make the respondent(s) feel more comfortable in the situation • The researcher should introduce themselves and the research • Discuss anonymity and confidentiality • Introduce respondent(s)

41. Qualitative Instruments • Wording Questions • Writing questions for a guideline is much like writing a questionnaire with all open-ended questions; therefore, many of the recommendations are the same, keeping in mind: • You want a logical, systematic conversation • Replicate as much as possible by other interviewers

42. Qualitative Instruments • Wording Questions Most importantly: • Your questions must encourage discussion: • Why? • How? • What do you think about…? • In what ways…? • What do you know about…? • e.g., What is your perspective on privatizing the airlines in our country?

43. Qualitative Instruments • Wording Questions Provide probes for the researcher to use if the initial question does not elicit sufficient discussion. For instance: • Question: What do you think are the major development challenges facing your country? • Probe: Why are these more pressing than other challenges? • Probe: Describe how the government is meeting these challenges (probe: resources, partnerships, etc.?)

44. Qualitative Instruments • Wording Questions • Occasionally, a close-ended question maybe necessary, just be sure to include follow-ups • e.g., Do you believe families should have to pay for their children’s education? Why/why not? • Although you want to encourage discussion, make sure your questions are sufficiently focused that respondents do not go too far astray • You should set approximate time allotments for each section to be sure the researcher covers all the issues • For certain questions, you may want to take notes on a board for the respondents to see; indicate this in the guideline

45. Qualitative Instruments • Order of Questions • As with quantitative questionnaires, use the funnel approach, proceeding from broad, general questions to more specific questions • Opening questions should be easy and non-threatening

46. Qualitative Instruments • Length of Guideline • Keep in mind, the objective of this research is to encourage discussion. You should not expect to get through a lengthy guideline • The shorter the guideline, the more you will be able to encourage a thorough discussion • Only ask questions that meet your specific research objectives. For each question, ask: • Is this question necessary? • Will this question further the relevant discussion? • Will it meet my objectives?

47. Qualitative Instruments • Administering the Guideline • A skilled moderator/interviewer is essential to gathering quality data • Confidentiality must be ensured • Consider not using political polling firms • Keep in mind how you will be administering the guideline (e.g., focus group, in-depth interview) when deciding on the wording and length of the guideline

48. Qualitative Instruments • Administering the Guideline Focus Group: • Professional moderators • Best for encouraging discussion of a range of respondents’ opinions • Guidelines should be short to moderate in length. To keep a lively discussion, focus groups should not go too long (about 90 minutes, depending on the culture) • Be careful not to include sensitive issues about which respondents will want to maintain confidentiality and may be reluctant to discuss

49. Qualitative Instruments • Administering the Guideline Focus Group: • Moderators should be instructed on what to do if the discussion goes too far astray • Moderators must also maintain objectivity, so that respondents do not feel that they should respond in a certain manner • Moderators must be fluent in the respondents’ language

50. Qualitative Instruments • Administering the Guideline In-depth Interview: • Professional interviewer • Best for a longer guideline, a guideline that does not lend itself to group discussion, or when a group format would not be appropriate with the desired respondents (e.g., elite government officials) • Confidentiality must be ensured • Should be done at appropriate times with interviewers fluent in the respondent’s language