Survey Research Experiment: Components & Importance of Questionnaires

1. Week 2Survey ResearchExperiment SURVEY RESEARCH Components – Questionnaires & Interviews

2. Surveys are the most Popular method for information gathering in Social Research • Why? • Versatile – appropriate for studying most issues • Efficient – appropriate for use with large populations, relatively low cost, quick, can use many variables, can be geographically diverse • Generalizability – appropriate for probability sampling- can generalize from sample to population. • Omnibus studies – good for large studies like General Social Survey

3. Problems in Conducting Social Research • Sampling, measurement and Design issues • Use of Survey as Design – Society is dynamic – survey is one shot deal • Two types of Data Errors in Survey 1. poor measurement of cases that are surveyed 2. omission of cases that should have been surveyed • Why? Individuals don’t respond, poor sampling frame, sampling error (characteristics of sample members don’t match the characteristics of population)

4. Survey Instrument Types • Questionnaire – Self Administered • Interview Schedule- Questions asked face-to-face and answers recorded by researcher. • Both must be well thought out, clear, and integrated as a whole with questions complementing each other Questions must be asked of many people, not just one person Must use same instrument for all participants Questions must be understood in the same way by all participants

5. Principles of Question Writingfor Survey research • Avoid confusion (double negatives and double-barreled questions) • Screening – filter questions to move participants to other parts of survey based on their answers (contingency questions) • Bias- Avoid words that trigger bias (loaded) i.e. communist take-over like Vietnam Instead of: in a situation like Vietnam

6. Don’t lead or mislead with questions • Response range deficient – response range must reflect full range of possible answers • Continuums – number of positive and negative categories must be balanced • Minimize fence sitting – neutral category may attract too many – only use neutral when you need to know how many fence-sitters you have.

7. Don’t Know/No Opinion – May omit if it keeps people from expressing their real opinion People float to no opinion categories However, forcing people to choose an answer even if they don’t know anything about the topic is also a problem Make sure you know why you are using these categories and have a purpose for them

8. Response Categories for Questions • Can be closed ended or open-ended • Closed-ended- Answer categories are provided by researcher. Respondents check or circle their choice – Were you abused? 1.yes 2.no • Open-ended – Respondent provides the answer in his/her own words. Can be lengthy and disorganized. Used to get at the meaning the subjects attach to their answers – How were you abused?_____

9. Best Response Category for Questionnaires • Closed Ended (Fixed-choice) (one and only one possible response for everyone who is asked the question) 1. Male 2. Female 3. Other • Categories must be exhaustive (if you can’t list all possible choices then provide an ‘other’ category • Categories must be mutually exclusive (Ranges of ages, incomes, years of school etc. should not overlap and should not leave out any value ranges) 1. 20-30 2. 21-40. • Check all that apply should be used with caution and kept to a minimum.

10. If a variable can’t be measured with one question then create an index or scale ( Several questions to measure the same variable) Depression can’t be measured simply by asking Are you depressed? If some questions in an index carry more weight than others then you have a scale

11. Guttman Scale • It is okay for Aliens to work where I work • It is okay for Aliens to live in my neighborhood • It is okay for Aliens to live next door. • It is okay to be friends with Aliens • It is okay to marry Aliens • It is okay to have children with Aliens

12. Multi-item Index • Several questions for one concept • Use statistical testing of index to make sure all questions fit in index (i.e. correlation or factor analysis) • Popular Likert Index (often called scale) Ordinal measure ranging from low to high. Is the subject liberal about ethnic differences? Aliens are bad 1. Strongly Agree, Agree, Disagree, 4. Strongly Disagree Can sum the responses for individual statements to get a summated rating scale for each subject

13. Design Issues • Pay attention to the order of questions (questions can influence each other) • Sort the questions into thematic categories • Start with easy and interesting questions • Maintain consistent focus on the research problem– eliminate irrelevant questions • Have a clear conception of the population to be sampled

14. Make the Questionnaire attractive • Indicate path through the questionnaire with words, arrows or graphics • Mailed questionnaires must include a cover letter and stamped, self-addressed return envelop

15. Response Categories and other issues for Interview Schedules • Looks like a Questionnaire, except it is read to the respondent by the interviewer • A mix of Open-ended and Fixed Choice • Interviewer negotiates through the questionnaire path for the subject • Interviewer must seem to be asking questions from memory and float seamlessly along • Interviewer must make the interview seem personal and must seem engaged and interested • Must begin with an introductory statement to draw the respondent in • May want to send a copy of questions to the interviewee before the interview appointment

16. Refine and Test Questions • All questionnaires and interview schedules must be pre-tested on a small sample • Focus groups can be used to formulate content of questions for surveys

17. Most Popular 5 Designsfor Administering Surveys • Mailed • Group-administered • Phone • In-person Interview • Electronic

18. Important Issues for Administering Surveys • Manner of Administration – Is the survey completed by the respondent or does the researcher ask the questions and record the answers • Structure – Is the instrument highly structured or relatively unstructured • Setting – Is the survey being answered in an individual or group setting • Cost – What are the costs and time expectations of each of the 5 basic designs?

19. Mailed, Self Administered Surveys • Respondents administer the survey themselves • Low response rates • Have to do follow up, include self-addressed and stamped envelope, maybe token reward • Mostly structured • Lower cost • Subjects must be literate • Can’t be sure who answers it

20. Group-Administered Surveys • Researcher distributes or provides access and it is administered in group setting • Individual Group members administer surveys to selves • High response rate • Mostly structured • lower cost • Works well with students, employees, members of military or institutionalized populations • Participants often feel coerced, less likely to answer honestly, Believe researcher is not independent of sponsoring organization

21. Telephone Surveys • Respondents interviewed via phone so is not self-administered but by professionals • May be computerized and responses automatically recorded • Can be structured or unstructured • Cost is high • Safe, efficient, large samples, fast turnaround • Validity questionable –may not reach proper sampling frame • Lots of incomplete responses

22. Most phone surveys use random digit dialing in sampling process • CATI (Computer-assisted Telephone Interviewing software) often used • May require multiple call backs • 10-15 time max • People have negative bias toward telemarketing • Impersonal nature of phone interviewing (need careful training) (Also could be computer asking questions)

23. In-Person Interview Surveys • Face-to-face interaction • More, Longer and more complex questions • Administered by professional • Mostly structured • Cost very high and time consuming • High response rates • Interviewer can monitor conditions • Interviewer can probe for meaning • Can use more open-ended Qs

24. Order in which questions read and answered can be controlled • But, in spite of flexibility, all respondents should have same interview experience • Interview should seem personalized • Computer-assisted personal interviewing software (CAPI) can be used. Can display interview schedule on laptop. The software processes the answers and checks to make sure answers fall into allowed ranges • Difficult to record answers and appear to be engaged

25. Electronic Surveys • Self-Administered via computer over Internet • Delivered via Web or Email • Mostly structured • Cost very low • Email surveys usually have to be short • Web-based surveys can be longer • Require programming expertise by researcher • Web-based can include graphics, links, pull-down menus, pictures, audio • Responses recorded directly into researcher’s database (eliminates data entry error & quick report generation) • Could be problems with sampling frame • Fast growing technique

26. IVR – (Less Popular)Interactive Voice Response System • Allows Internet surveys to be achieved with telephone-based technology • Respondent received automated call • Answer questions by pressing numbers on touch-tone phone or speaking numbers that are interpreted by computerized voice recognition software • Is a very impersonal approach

27. Mixed Mode Surveys • Combinations of 5 basic survey administration designs (group,mailed, interview, phone, electronic) • Maximizes quantity and quality of data • Allow strengths of one design to compensate for weaknesses in another • Maximizes likelihood of securing data from different types of respondents

28. Comparisons • Mailed surveys – worst response rate best for large, diverse populations inexpensive good for sensitive issues Best alternative – Phone survey

29. In-person Survey Best in terms of possible length and complexity Not as good for sensitive questions Require a lot of training Best response rate Expensive and Time consuming

30. Electronic Surveys Not everyone has Internet connection or capacity to display complex Web pages Conclusions: • In-person Interview is strongest design and generally preferable • Phone surveys have many of the advantages of in-person interview but at much less cost. Response rates a problem. • A decision about the best survey design must take into account the unique features and goals of the particular study. • Survey research which is mostly Quantitative can be enhanced by combining with Qualitative research such as Case Studies, Focus Groups or Observation

31. Ethical Issues in Survey Research • Fewer ethical issues than Experiment or Participant Observation • In group-administered survey the question of participation being voluntary arises • Primary ethical concern is Confidentiality Answers to questions could be damaging to subject if disclosed Only research personnel should have access to information that could link subjects identity to answers. Use ID numbers not names Anonymity – But, if you have no identifying info linking subjects to their surveys then follow-up is not possible

32. EXPERIMENT IN RESEARCH • Answers questions about the effect of a treatment or intervention on some other variable whose values can be manipulated by the researcher. If Clarinex given then symptoms reduced. 1. Clarinex given 2. placebo given 3. nothing given • Experiment is powerful design for testing hypotheses – (A research question in testable format) • Hypothesis is explanatory - quantitative

33. Hypotheses • A specific expectation deduced from a more general theory • We test the hypotheses not the more general theories • Variation in one variable in the hypothesis is supposed to predict, influence or cause variation in another variable in the hypothesis

34. Example • Deterrence Theory – Punishment deters crime • Deterrence Hypothesis – If abusers are arrested on first offense then abusers will be less likely to abuse again. or As arrests for abuse increase, recidivism for abuse decreases

35. Components of Hypotheses • Independent variable – considered to be the causal variable. It is manipulated by researcher to effect (influence or cause) change in Dependent (another) variable • Independent variable=Cause • Dependent variable=Effect

36. CAUSATION • A cause is an explanation for some characteristic, attitude or behavior • Causation is often shown by experiment – A variation in the IV results in variation in the DV

37. Essential Components of Experimental Design • At least two comparison groups of subjects (usually experimental and control groups) • Variation in the independent variable occurs before assessment of change in the dependent variable • Random assignment to the two (or more) comparison groups • A combination of these components gives us more confidence in the validity of causal conclusions

38. Confidence is increased with the addition of two other components: 1. Control over the context of an experiment 2. Identification of the causal mechanism

39. Conditions of a True Experiment • Must have at least one experimental group (subjects who receive some treatment or manipulation) • At least one comparison group (subjects to whom the experimental group can be compared) who receive a different treatment or no treatment • If no treatment is given to the comparison group it is called a control group

40. All true experiments have a post-test (measurement of the outcome in both groups after the experimental group has received the treatment) • Does not require a pre-test but it can be advantageous Pre-tests provide a measure of how much the experimental and comparison groups change over time.

41. Popular Pre-test/Post-test Control Group Design • Two or more groups (at least one experimental and one control) • Pre-test and post-tests • Random Assignment to Groups (Randomization) Note: This is not the same thing as Random Sampling to be discussed in Sampling Lecture

42. Random Assignment places pre-designated subjects into two or more groups on the basis of chance If comparison group differs from the experimental group in anyway besides not receiving treatment or receiving a different treatment, can’t determine for sure what the unique effects of the treatments are.

43. In true experiment subjects must be randomly assigned to the comparison and experimental groups. Eliminates systematic bias • When used, the odds of a difference between the comparison and experimental groups by chance can be calculated. • Larger the group the better this works (over 30 subjects is best)

44. Matching – subjects are assigned to the different groups based on similarity of variables such as gender, age, year in school, or other important characteristics • researcher identifies in advance all the important variables on which to make a match of the assignment to groups (should be used in conjunction with random assignment – not instead of

45. Limitations of True Experiment • Difficult to isolate the actual mechanism by which treatments have their effect • Difficult to guarantee that control has been maintained (more likely to maintain control in lab than in field)

46. Quasi-Experiments • Often testing a hypothesis with a true experimental design is not feasible • It may be too costly, time consuming or desired setting or subjects not available • In Quasi-experiment- subjects in groups may not be randomly assigned

47. Popular Quasi-Experiment Designs • Nonequivalent Control Group – Experimental and control groups are designated before the treatment occurs, not by random assignment • Before and After Designs – Has a pretest and posttest, but no comparison group. The subjects exposed to treatment serve as own controls by comparison of pre and post tests 1. Multiple Group Before and After Designs– several before and after comparisons are made involving the same variables but different groups and then the groups are compared 2. Repeated Measures panel Designs- the same group is observed many times (30 or more), receiving many pre and post tests. Why? To study the process by which an intervention or treatment has an impact over time.

48. Identifying Cause • Quasi-Experiments only partially meet the criteria for identifying a cause. • But, the association between the IV and DV and the Context in which change occurs can still be met in this type of experiment

49. Validity in Experiment • Internal Validity 1. Selection bias – characteristics of treatment groups differ (attrition maybe) 2. Endogenous change – subjects develop or change during the experiment (maybe experiment effect) 3. History – something occurs during treatment which influences results 4. Contamination – one of the groups is aware of the others and is influenced by this knowledge

50. 5. Treatment misidentification – researcher is not aware of some process that affects treatment . For example the way staff administers treatment. Hawthorn Effect – patients perform better because feel special. In Evaluation Research, clients know research findings may affect chances of further funding Double Blind procedures can offset this (staff doesn’t know who is getting what) • Placebo Effect- patient improves on placebo because they think they are getting the real thing