Research Methods

1. Research Methods Survey Research

2. Overview • Survey research is the method of gathering data from respondents thought to be representative of some population, using an instrument composed of closed structure or open-ended items (questions). • This is perhaps the dominant form of data collection in many areas of behavioral research, providing for efficient collection of data over broad populations, amenable to administration in person, by telephone, and over the Internet. • Some forms of survey research by telephone or the Internet may be completely automated. • Critics of survey research methodology hold it to be a method which artificially forces respondents to formulate opinions, masking the complexity of conflicting views and unconscious biases within each respondent, and critics note that in many arenas (e.g., race relations) survey items poorly predict actual behavior.

3. Key Terminology and Conceptual Stuff • Survey Instrumentation. • The survey "instrument" is the schedule of questions or response items to be posed to respondents. • Items • Items are the individual survey questions or statements for which a response is solicited from the respondent. • Interviews and questionnaires. • Ordinarily, "interviews" refers to the face-to-face administration of a survey instrument, whereas "questionnaires" refers to mail or other indirect methods of administration. • Response structure • Response structure refers to the format of items, which may be open-ended or structured. • Structured items may be multiple choice, multiple response, dichotomies, rank lists, or a variety of other formats. • Survey model. • A social science model is can be depicted graphically as a set of variables (represented by circles) connected by a set of causal effects (represented by single-headed arrows) and correlations (represented by double-headed arrows). • The survey should, at a minimum, include items pertaining to each circle and arrow in the model. Multiple items to measure each variable and relationship are preferred.

4. Survey order. • Typically, after an introduction which discloses the sponsorship of a survey, a survey begins with non-threatening items which arouse interest. • The first question should be clearly related to the announced purposes of the survey (not a background item, for instance). • Some recommend the second question be open-ended, to allow the respondent to "get into" the subject. • Non-threatening background information questions (e.g., demographic information) should be posed early so these controls will be available if the respondent fatigues and does not answer all the later items. • But …There are others who say this should be at the end. • The survey then proceeds to attitude questions, often sequencing from general and less threatening items toward more specific and more sensitive items. • Sensitive background items, particularly the income item, are usually put at the end. • However, the more toward the end of the survey an item is, the lower its response rate is apt to be. • For this reason, balancing the foregoing considerations, it is desirable to administer the survey with different forms with different orders so as to lessen the order/response bias.

5. Filter items • Filter items are ones which allow the elimination of unqualified respondents during post-processing (e.g., respondents who lack minimum information to respond intelligently). • Cross-check items • Cross check items are internal check items which test consistency with other responses. • Asking age at one point and date of birth at another would be a pair of cross-checked items. • An extension of this is the split-form interview, which might be administered to a both partners in a marriage with a view toward cross-checking for consistency. • Random probe items • Random probe items of an open-ended nature are randomly interspersed by item and by survey form such that any one respondent is only probed on a few items. • Probe items are "Could you tell me a little more about what you mean?" or "Could you give me an example of what you mean?" • Probe responses are used to better understand the meaning of frequency distributions of responses to the corresponding structured items.

6. Item bias. • Bias may creep into survey items in a variety of ways. • Ambiguity. Questions should be specific, avoiding generalities. • E.g.: "On a scale from 1 to 10, how popular is President Bush at this time?" This example begs the question, "popular with whom?" • Some respondents will answer the item in terms of popularity with themselves, • others will answer in terms of popularity with their peers, • and yet others will respond in terms of their recollection of past opinion polls. • Items often need to be anchored in one or more of the familiar journalistic dimensions: who, what, when, where, why, and how? • If it is possible for respondents to interpret questions in dissimilar terms, they will. • "Have you participated in a political rally?" is also ambiguous, for instance, because participation is undefined and some will answer affirmatively due to passing within earshot of a political speaker, while others will answer negatively because, while they attended rallies, they were not the organizers. • As a third example, "Do you favor government policy toward crime control?" leaves ambiguous the level of government (local, federal, state) as well as just what dimension of law enforcement is at issue. • As a fourth example, when value preferences are requested by ranking "above average," "average," "below average," etc., ambiguity is introduced unless the question is answered, average in comparison with what? • In general all pronouns, nouns, and noun-phrases should have a clear referent.

7. Mutual exclusivity. When multiple response items are allowed, bias is introduced if the items are not mutually exclusive yet only a single item may be selected. • Non-exhaustive response set. Bias is introduced when the response alternatives available to the respondent leave out valid choices they would otherwise make. • The most common example is leaving out such responses as "neutral" or "don't know" when, in fact, respondents may well be neutral or may actually not know, rather than be hiding their "true" responses which the researcher is trying to force out by omitting these categories. • Ambiguous terms, such as "parents," can also lead to non-exhaustive response sets, as by omitting the distinction between birth parents and adoptive parents. • Likewise, "single" may omit the distinction between "never married," "widowed," and "divorced." • Rank lists. Ranking can be a challenging task. • Many survey researchers recommend that respondents not be asked to rank more than four or five items. • Beyond that, respondents may give arbitrary rankings just to get past the item.

8. Loaded terms. E.g., "Do you lean more toward the pro-life or toward the pro-abortion position on issue of termination of late-term pregnancies where the health of the mother is threatened?" • This example is biased because one position is labeled with its most favorable label (pro-life, rather than anti-abortion), while the other position is labeled with its less favorable label (pro-abortion, rather than pro-choice). • As another example, more people will "not allow" something than will "forbid" the same thing. • Leading questions. e.g., “As a small business owner, do you favor an increase in the federal minimum wage to $8.00?" • This example is not as leading as an item of the type, "You favor X, don't you?" but it is still leading because it does not legitimize both affirmative and negative responses. • A negative response may appear stingy or mean-spirited toward the poor, and this may bias the survey toward too many affirmative responses. • A better method is to phrase such policy options in the form, "Some people favor X, while other people are opposed to X. What is your opinion?"

9. Recall items. People's ability to recall the past is limited. • The more current and specific the question reference, the better. • If recall is necessary, the time frame should be as recent as possible and not over six months unless the reference is to major events (e.g., marriage, changing jobs, buying a car). • Unfamiliar terms and the use of jargon. e.g., "Do you consider environmental regulation of wetlands to be an invasion of the sovereignty of the states?" • Terms such as "sovereignty" are apt to not be well understood by typical survey populations. • When a term not in popular usage must be used in an item, such as an item asking opinions about the Civil Rights Act of 1964, the interviewer must precede the item with a brief explanation. • Wherever possible, familiar terms should be substituted for unfamiliar terms.

10. Requiring inaccessible information. An item may use familiar terms but require information most respondents would not know. • For instance, a question about Stalin's treatment of the Cossacks might have been acceptable long ago, but today's population of respondents is apt to know little about this subject and perhaps not even recognize "Stalin." • As a second example, the item "Is your family income above, about equal to, or below the official poverty rate for a family of four?" is flawed, among other reasons, because people are not apt to know what dollar amount is the referent today, making the item unacceptable for a determination of fact (although possibly acceptable for a determination of self-perception). • Multidimensionality. A form of ambiguity arises when items are multidimensional. E.g.: "On a scale of 1 to 10, please rank the performance of the president?" • The respondent may be torn between multiple dimensions: personal vs. official performance, or domestic vs. foreign policy performance, for instance. • Example two: "Do you believe President Clinton lied under oath in the Monica Lewinsky scandal, impairing his effectiveness as president?" • This item calls for two opinions, not one: * one on lying under oath and * one on subsequent effectiveness. It should be broken into two separate items.

11. Compound items. Items with compound clauses may not be multidimensional but may involve undue complexity (see below). • For example, the item, "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months and which has limited the kind of work you could do on your job?" is better broken into two items: (1) "Do you have or have you every had a physical, mental, or other health condition which has lasted over six months?" and the follow-up item, (2) "If you answered yes to the previous question, did this condition limit the kind of work you could do on your job?"

12. Complexity and memory overload. • It is possible to overtax the respondent by requiring an excessive memory burden. • The more complex the item, the easier it is to overload memory. • The explanation should not demand the respondent learn complex new material. • Simply using double negatives in an item may introduce undue complexity. • If there are over five alternatives, a show card should be used to allow the respondent to view the alternatives, not simply hear them orally in an interview situation. • Poor grammatical format. • Weak grammatical format can introduce bias. • For instance, the item, "Would you say that you approve very strongly, strongly, ..." presents "dangling alternatives" which the respondent must memorize before even knowing what the question is. • This format is frustrating and may bias responses toward the first-presented response, or toward negativity. • Putting the subject first is the preferred order.

13. Hypothetical items. • Hypothetical items (e.g., "What would you do if ...") creates a difficult challenge for respondents. • Seriously considering such items requires time for imagination and consideration. • People tend to base responses to such items on their most-related actual experiences, and it may be better to ask about such experiences directly. • Inappropriate assumptions. • Items should not contain false or arguable premises. The respondent should not be required to make a false or arguable assumption in order to respond to an item on its face. • For instance, "How much improvement in your life would passage of the Equal Rights Amendment make? A great deal, some, very little, or none?“ • …is an item which assumes that the effect of ERA passage could not be negative on a respondent, forcing respondents who believe the effect would be negative to either skip the item or give a response which does not represent their views. • Mismatched item and response set. • The response categories should be appropriate to the dimension probed by the item. • For instance, "How unusual do you think it is for a person to donate $100 or more to a presidential candidate? • A great deal, some, very little, or none?" is a mismatch because "a great deal unusual," "some unusual," etc., are not grammatically acceptable responses and will confuse respondents.

14. Survey Error. • Survey error, over and above item bias, survey error includes such factors as faults in sampling, coding, tabulating, data processing, interviewer bias, researcher bias, and data misinterpretation.

15. Pretesting is considered an all-but-essential step in survey research. • No matter how experienced the survey researcher, pretests almost invariably bring to light item ambiguities and other sources of bias and error. • In fact, Converse and Presser (1986) argue cogently that a minimum of two pretests are necessary, with pretest sizes of 25 - 75 administered to respondents similar to those who will be in the final sample. • The first pretest may have up to twice the number of items as the final, as one purpose is to identify weaker items and drop them from the survey. • Items may also be dropped if the first pretest shows they exhibit too little variance to be modeled. • The first pretest will also have many more probe items that the final, and respondents may even be told they are being given a pretest and their help solicited in refining the instrument. • Other purposes of the first pretest are • to see if respondent interest is aroused, • if respondent attention can be maintained, • if interviewers and respondents feel the survey has a natural flow, and • if interviewers experience problems such as need to repeat questions, need to correct misinterpretations of questions, need to handle volunteered information, sections where the respondents wanted to say more, or questions which seemed sensitive. • The first pretest results, hopefully, in a much-changed, finished instrument. • The second pretest is used simply for polishing, trimming, rearranging, and other refinements, but not adding new dimensions or making major substantive changes in the survey. • If such changes prove necessary, a third pretest will also be needed.

16. Split sample comparisons. • After pretesting, if there is any remaining doubt about the reliability of one or more items, the researcher should consider split sample comparisons, where two versions of the same item appear on two different survey forms administered randomly. • If the mean responses of the comparison subsamples differ significantly, the inference is made that the two versions represent two different items, not two versions of the same item.

17. Stuff to Consider When Surveys and Their Information are Critical • Okay, How close a correspondence is there between a person’s opinion survey responses and their actual actions (behavior)? • Basically this question is …”Is there an attitude-action gap?“ • In some areas, such as voting, there appears to be a close correspondence between how respondents say they will vote and how they actually do vote. • In other areas, such as controversy-related areas (e.g., race-related attitudes), the discrepancy between what people say and what they do appears to be substantial. • Tartar (1969) found 65% of respondents changed their behavior away from surveyed attitudes when confronted with actual situations requiring behavioral response. An early survey by Ehrlich (1969) concluded, • "Studies on the relation of attitudes and behavior have almost consistently resulted in the conclusion that attitudes are a poor predictor of behavior." Ehrlich gave seven reasons for this gap: • There may be no clear way for an attitude to become expressed in a behavior. • Attitudes may reflect fantasy or sublimation more than realistic responses. • Individuals may not be willing to disclose honest expression of certain opinions. • Individuals may translate opinions into action in unpredictable ways, such as suppression and inaction, displacement into other arenas, and rationalization. • Individuals may lack the informational resources to act. • Individuals may lack the opportunity to act. • Individuals may lack the sense of efficacy to act.

18. In would like to use open-ended items to enrich the data I receive. So, what are the pros and cons of using open-ended items? • Open-ended items are frequently used as complements rather than alternatives to structured items. • As a follow-up to a structured item, an open-ended item can bring to light unanticipated interpretations and variations in the meaning of responses. • For instance, in a ranking item about job satisfaction, a structured response may show "income" to be the most important satisfaction factor, but a follow-up open-ended item may show some respondents mean "high income," whereas other respondents mean "steady income." • However, open-ended items are not necessarily more accurate. • An open-ended question may elicit one range of preferences, yet when other choices are presented in a structured question, respondents may well alter their open-ended preferences to re-rank in the light of previously-unconsidered alternatives in the structured item. • Ideally, one would use both, with an open-ended probe question after every structured item. • This, however, would be prohibitive in time and fatigue costs of the interview, which is why random probe questions are used, as discussed before.

19. Is it okay to force a choice, or is a “no opinion” option necessary? • It is very tempting to leave out “no opinion,” “not sure,” and “I don't care” options in structured questions. • Forcing a choice not only yields the direction of preference, but also requires the respondent to give a more considered response. • However, forcing a choice does distort the responses of individuals who genuinely have no opinion, are genuinely unsure, or who genuinely do not care. • When the researcher can assume everyone really has an opinion, then forced choice is justified. • If not, the researcher should retain the no preference options, or should consider filter and intensity questions. • Filter questions precede an item and are of the type, "Here is a statement about X. Not everyone has an opinion on this. Here is the statement: ... Do you have an opinion on that?" • If the respondent answers “Yes” to the filter question, then a forced-choice item is presented. • Intensity items follow forced choice items and ask “How strongly do you feel about that -- extremely strongly, very strongly, ...?" • A separate intensity item is needed because extremity on the forced choice item is not equivalent to intensity: one may hold an extreme position non-intensely and may intensely hold a middle position. • In using forced choice rather than offering a “no opinion” option, the researcher is assuming that those who would have given “no opinion” responses will be low intensity on the intensity item, but not necessarily in the middle on the forced choice item itself.

20. Fox, J. A., & Tracy, P E. (1986). Randomized response: A method for sensitive surveys. Thousand Oaks, CA: Sage Publications. Quantitative Applications in the Social Sciences Series No. 58. • Here’s one you may or may not ever worry about … “How can I get a more valid response to highly sensitive questions?” • This issue is treated in depth by James Fox and Paul Tracy (1986). • The randomized response strategy taking the “heat” off of subjects by mixing randomized responses with real ones. • The basic randomized response method is like this… • Consider the question "Have you ever used an marijuana?" • Respondents can be told to flip a coin or use some other randomizing device, then if heads appear (not seen by the interviewer) respondents are told to answer the question honestly but to answer "Yes" regardless if tails appear. • If the result for a survey is 60% "Yes," we can reason that the true response rate is 20% (half of true "Yes" responses will also be part of the 50% expected tails-determined "Yes" responses and half not, so the true percentage is twice the excess of the observed response [60%] less the random response [50%]). • Fox and Tracy extend this to polytomous and quantitative responses and discuss various refinements as well as alternative approaches which have proved problematic.

21. Gelman, Andrew, Gary King, and Chuanhai Liu (1998). Not asked and not answered: Multiple imputation for multiple surveys. Journal of the American Statistical Association, March, 1998. • Problems of Return & Completion • Andrew Gelman, Gary King, and Chuanhai Liu (1998) have presented a method of analyzing a series of independent cross-sectional surveys • in which some questions are not answered in some surveys and • some respondents do not answer some of the questions posed. • The method is also applicable to a single survey in which different questions are asked, or different sampling methods used, in different strata or clusters. • The method involves multiply-inputting the missing items and questions by adding to existing methods of imputation designed for single surveys a hierarchical regression model that allows covariates at the individual and survey levels. • Information from survey weights is exploited by including in the analysis the variables on which the weights were based, and then re-weighting individual responses (observed and imputed) to estimate population quantities. • They also develop diagnostics for checking the fit of the imputation model based on comparing imputed to non-imputed data. • Their example that motivated this project --- a study of pre-election public opinion polls, in which not all the questions of interest are asked in all the surveys, so that it is infeasible to impute each survey separately.

22. Behling, O., & Law, K. S. (2000). Translating questionnaires and other research instruments: Problems and solutions. Thousand Oaks, CA: Sage Publications. Series: Quantitative Applications in the Social Sciences No. 133. • Issues of Cross-Culturalism and Multiple Languages • Here the question is simple … “How do I handle the need to translate items across languages, yet have the same meaning? “ • Simple direct translation of validated U. S. scale items into another language often will not create a valid scale in the other language. • One must consider three dimensions of the problem: • semantic equivalence, • conceptual equivalence, and • normative equivalence of items. • An item may be acceptable in one dimension but not in another. • Semantic equivalence is sought through the translation/back translation method, having independent translators translate from one language to another, and then back again, to see if the original and re-translated item remain the same. • Conceptual equivalence may be sought by such methods as factor analysis, demonstrating that items in one language version have the same factor structure (load similarly on underlying factors) as in another language version. • Normative equivalence requires such strategies as developing close relations with respondents, using trusted agents for interviewing, providing assurance of anonymity or confidentiality, and pilot testing extensively. • Orlando Behling and Kenneth Law (2000) provide numerous additional practical suggestions and insights into the translation problem and this text is strongly recommended for those who will face translation issues.