Mastering Survey Research: Questionnaire Preparation Guide

1. CHAPTER 9 Survey Research

2. Basics • In Survey Research, the researcher selects a sample of respondents and administers a standardized questionnaire. • Main problem: • How to prepare a questionnaire and how best to administer it to respondents. • Surveys are social telescopes, best method for doing structural research or describing populations too large to observe.

3. Attitudes and Dispositions • Surveys are good guides for measuring attitudes and dispositions of a large population. • The accuracy of surveys or ‘polls’ is in the representativeness of the sample and whether questions are formatted properly or not.

4. Guidelines for asking Questions • Operationalization that we talked about earlier involves asking people questions to measure what actually indicates the concept you are measuring. • The Questionnaire is the instrument (like a thermometer) designed to MEASURE accurately/scientifically what we desire to measure- it generates data that can then be analyzed.

5. Question Format • There are two types of questions • Open-ended and Closed-ended • In open-ended questions, the respondent is asked to provide his or her own answers to the question (requires coding before processing) • In closed-ended questions, the respondent is asked to select an answer from among a list provided by the researcher (often just need to be fed into the computer, but a disadvantage is that they might miss important aspects of responses if the respondent is restricted by researcher choices)

6. Closed-Ended Questions • An example is the Likert Scale • A format in which respondents are asked to strongly agree, agree, disagree, strongly disagree etc- it measures the extent to which respondents hold a particular attitude. • Closed-ended questions have two structural requirements: The choices should be 1) Exhaustive 2) Mutually Exclusive.

7. 1. Exhaustive choices means that your choices should include ALL possible responses that might be expected. • 2. Mutually exclusive implies that only one choice can be logically selected by the respondent, one choice excludes the possibility of the other choices.

8. I. Making Questionnaire Items Clear. • Questions and possible responses should be 1) worded clearly and 2) should be unambiguous in meaning. • In other words, the questions and response categories (if closed-ended) should be clear enough so as not to elicit further questions from the respondent. • E.g. if you are measuring unemployment which you have conceptualized as being an active search for work by someone currently not employed, in the past 4 weeks, you have to clearly define what you mean by the “past 4 weeks” when the period starts and when it ends. What active search means and what ‘not employed’ means

9. II. Avoid Double Barreled Questions • Double barreled questions are those that require one answer to a question that actually has multiple parts: • E.g. Should the U.S. abandon its space program AND spend that money on healthcare. • Whenever AND comes in the question beware of the double barreled question. Also the second part is LEADING the person to answer in a specific way by giving you a choice of what is to be done with the money, so it is choosing your response to the first part.

10. III. Respondents Must be Competent to Answer. • Asking a 40 year old to remember specific details about their childhood - the person will probably not remember accurately, i.e. he or she is not competent to answer your question. IV. Respondents must be Willing to Answer • People are reluctant to reveal viewpoints that they think might be a minority opinion or might be unpopular. In those circumstances what can you do to get them to give an honest response?

11. V. Questions should be relevant Know your audience or you might get responses that are irrelvant. VI. Short Items are the best Can be read quickly and answered easily. VII. Avoid negative items – these can confuse the respondent, people tend to read over the word “not.” VIII. Avoid biased items and loaded terms: Bias in this case refers to a question that encourages a particular response, e.g. “Don’t you agree that…..?” is a leading question. Loaded terms would seek through the avenue of “social desirability” a certain response from the respondent, e.g. based on the congruence of a particular response with a well known culturally positive value.

12. Questionnaire Construction The format of a questionnaire is as important as the wording of the questions. • The questions should be spread out and uncluttered. • Closed-ended questions can be constructed in numerical form for easy processing • E.g. Have you ever been a victim of racial discrimination? 1. Yes 2. No 3. Don’t Know • Contingency Questions: Answer to one determines if the other questions are relevant to you. • E.g. If married answer questions 13-20, otherwise go to question 21.

13. Matrix Questions: • Same set of answer categories for multiple questions- saves space and ensures ease of answering by the respondent. e.g. SA A D SD Uncertain • Q1 • Q2 • Q3

14. Ordering of questions • Earlier questions might affect later answers • Put the most important questions to your study first. Questionnaire instructions: • Help the respondent make sense of the questionnaire • Let them know if you want a single answer because sometimes more than one answer might appeal to them • Some questions require special instructions, make sure those are provided, like instructions that go with contingency questions.

15. Pretest the Questionnaire • To see how respondents feel and any errors that stand out • Also use ‘cognitive interviewing’- ask the test respondents for their comments on the questionnaire itself. • Self Administered Questionnaires- mailed to respondents, monitor returns (devise a way to check who returned), send follow up mailings, only to non respondents, after a reasonable amount of time.

16. Response Rate • Most important in surveys- affects the representativeness of the sample • Calculated as : • The number of people participating in a survey (divided by) The number of people selected in the sample • Also called the completion rate or the return rate.

17. Interview Surveys • Researchers send interviewers to ask questions orally and record respondents’ answers • Higher response rate • Decreases the number of ‘don’t knows’ • Interviewer can clarify matters • Can observe as well as record • Disadvantage- the Hawthorne effect.

18. Guidelines for Survey interviewing • Dress somewhat similar to the people you will be interviewing- dress indicates “attitudes and orientations.” • Be familiar with questions and question specifications should be prepared beforehand if possible. • Be familiar with probing techniques (i.e. requesting the respondent for clarification) • Interviewer should be trained beforehand in the above, and in administrative guidelines and procedures for recording the answers.

19. Other survey methods • Telephone surveys, through random digit dialing (RDD), not everyone has a land line or a cell phone- what does that do to the sample representativeness? Response rates of telephone interviews: Around 60% (1996 figures)- what does this do to sample representativeness? The GSS (General Social Survey) uses personal interviews and their response rate is around 70%.

20. Strengths and Weakness of Surveys • Make large samples possible. Indispensible for structural analysis. • Objectivity and reliability- given standardized questions, ease of analysis. Weakness: • Decontextual at times, not too good for micro analysis, misses depth that can create validity problems • Standarization means things get lost in the ‘translation.’

21. Secondary Data Analysis • Data processed and collected by one researcher are reanalyzed for a different purpose by another. E.g. Sources of secondary data 1)the GSS (general social survey) by the NORC (National Opinion Research Center) at the University of Chicago. 2) The Roper Center for Public Opinion Research, at the University of Connecticut. • Ethical Issues with Survey Analysis: • No harm psychological or otherwise (Job, relationships etc.) should come to the respondents, since they reveal info about themselves, it can cause harm.

22. In class Assignment • Using the variables that you will examine in your term paper, construct a six question survey with closed ended questions that will help you answer your “why” question. • Questions need to be clear, non-leading, concise and numbered for quantitative analysis.