Survey research

1. Survey research Dr. Yan Liu Department of Biomedical, Industrial and Human Factors Engineering Wright State University

2. Introduction • What Is Survey Research • Methodology of gathering data from participants (respondents) thought to be representative of some population, employing questionnaires or interviews • Advantages • Relatively inexpensive (especially self-administered surveys) • Useful in describing the characteristics of a large population • Provides an efficient methodology for asking people to tell about themselves • There is flexibility at the creation phase in deciding how the questions will be administered • Face-to-face, telephone, mail, email, internet, etc. • Standardized questions by enforcing uniform definitions upon the participants • Make measurement more precise • Ensure that similar data can be collected from groups then interpreted comparatively

3. Introduction • Disadvantages • Standardization forces developing general enough questions • Miss what is most appropriate to many respondents • Inflexible design • Require the initial study design (the tools and administration of the tools) to remain unchanged throughout the data collection • The researcher must ensure that a large number of the selected sample will reply

4. Constructing Questions to Ask • Define the Objectives • The survey questions must be tied to the research questions that are being addressed • Types of Questions • Attitudes and beliefs • Focus on the ways that people evaluate and think about issues • e.g. “Are you satisfied with this web design?” “Should students be allowed to bring laptops to the classroom?” • Facts and demographics • Ask people to indicate things they know about themselves and their situations • Demographic information is necessary to adequately describe the sample • e.g. age, gender, ethnicity, income, employment status, certain technical skills, etc. • Behaviors • Past behaviors or intended future behaviors • e.g. “Do you plan to purchase a computer within the next 3 months?” “How many computers did you purchase in the last five years?”

5. Constructing Questions to Ask • Question Wording • Simplicity • The questions should be relatively simple • Avoid jargon or technical terms • Give clear definitions and explanations to the possibly unfamiliar terms or concepts • Avoid “double-barreled” questions • A double-barreled question asks two or more issues at once • e.g. “Is it important that an online shopping website be esthetically pleasing and provide detailed information about the company?” • Combining more than one query into one question makes it unclear which attitude is being measured • Avoid “loaded” or “leading” questions • A loaded question is one that contains loaded or controversial words • e.g. Politicians often avoid the word “environmentalist” because it creates a negative reaction in some people regardless of the content of the statement • A leading question is phrased in such a way that suggests to the respondent that the researcher expects a certain answer • e.g. “Do you like a website that has distracting flashing banners?”

6. Constructing Questions to Ask • Question Wording • Avoid negative wording • Negative wording can confuse people and result in inaccurate answers • e.g. Instead of asking “Do you agree that students should not be allowed to bring laptops to the classroom?”, a better format would be “Do you agree that students should be allowed to bring laptops to the classroom?” • Use both mutually exclusive and collectively exhaustive response categories for closed-ended questions • Categories are mutually exclusive when there are no overlaps between them • e.g. “What is your current age”? A. 20 or less B. 20 to 30 C. 30 to 40 D 40 or above • Categories are collectively exhaustive when there is a category available to all potential responses • e.g. “How many computers have you bought during the last five years?” A. one B. two C. three D. four

7. Constructing Questions to Ask • Question Wording • Reverse the wording in some of the questions to help prevent “response sets” • A response set is a tendency to respond to all questions from a particular perspective regardless of their content • One way to detect the response set is to rephrase the wording in some questions so that consistent agreement is unlikely • e.g. “I prefer purchasing flight tickets online to going through travel agency to get my ticket” and “I always get my flight tickets through some travel agency”

8. Responses to Questions • Closed- Versus Open-Ended Questions • Closed-ended questions • A limited number of response alternatives are given • More structured • Easier to analyze • Useful when the dimensions of the variables are well defined • Open-ended questions • Respondents are free to answer in any way they like • Require time to categorize and analyze • More costly • Useful when the researcher wants to know what people are thinking and how they naturally view their world

9. Strongly Disagree Strongly Agree Disagree Undecided Agree Responses to Questions • Number of Response Alternatives • In public opinion surveys, a simple “yes or no” or “agree or disagree” dichotomy is often sufficient • In most research, it is often preferable to have a sufficient number of alternatives to allow people to express themselves • e.g. five- or seven-point Likert scale question ranging from “strongly disagree” to “strongly agree” • Labeling Response Alternatives • Only endpoints on the rating scale are labeled; the respondents decide the meanings of the other response alternatives e.g. Strongly Disagree Strongly Agree • Each rating has a clearly defined label e.g.

10. Finalizing The Questionnaire • Formatting the Questionnaire • The final questionnaire should appear attractive and professional • It should be neatly typed and free of spelling errors • Respondents should find it easy to identify the questions and the response alternatives to the questions • If a particular scale format is used, use it consistently • Carefully consider the sequence in which the questions are asked • It is generally best to ask the most important and interesting questions first to capture the attention of the respondents and motivate them to complete the survey • It is a good idea to group questions together when they address a similar theme or topic • Refining Questions • Before actually administering a survey, it is a good idea to give the questions to a small group of people and have them “think aloud” while answering them • How they interpret each question and how they respond to the response alternatives • Provide valuable information for improving the questions

11. Administering Surveys with Questionnaires • Overview • The questions are presented in written format and the respondents write their answers • Advantages • Less costly than interviews • Allow the respondents to be completely anonymous as long as no identifying information is asked • Disadvantages • Require that the respondents be able to read and understand the questions • Many people may find it boring to sit by themselves reading questions and providing answers • Types of Questionnaires • Mail surveys • Personal administration to groups or individuals • Internet surveys

12. Administering Surveys with Questionnaires • Mail Surveys • Surveys are mailed to individuals at home or business address • Advantages • Very inexpensive way of contacting people who are selected for the sample • Possible to reach the sample from a large population • The participants are able to work on the surveys at their leisure • Disadvantages • Low response rate (usually a little over 20%) • The researcher is not present to help if the participants have questions about the surveys

13. Administering Surveys with Questionnaires • Personal Administration • Researchers personally distribute questionnaires to groups or individuals • e.g. a classroom, conference room, cafeteria, etc. • Advantages • Relatively high response rate • The researcher is present so people can ask questions if necessary • Disadvantages • Not a good method for surveys that would benefit from a large population • Only useful when calling for very specific information from specific groups • Takes considerably more time than mailed surveys

14. Administering Surveys with Questionnaires • Internet Surveys • Questionnaires are administered on the internet • The respondents fill out the questionnaires online and the responses are sent to the researcher immediately • Sampling • Surveys are listed on search engines so people who are interested in a topic can discover that someone is interested in collecting data • Some of the major polling organizations are building a database of people interested in participating in surveys • Ask volunteers from internet special interest groups • Members use newsgroups, email discussions, bulletin boards, and chat rooms to exchange ideas and information

15. Administering Surveys with Questionnaires • Internet Surveys (Cont.) • Advantages • Easy to reach the sample from a large population • Easy to obtain samples of people with particular characteristics • Less expensive to send questionnaires online than to pay for postage or interviewers • Questionnaires can be delivered to recipients in seconds rather than in days as with traditional mail • Higher response rate than mail surveys • Respondents may answer more honestly with electronic surveys than with paper surveys or interviews • Disadvantages • There is ambiguity about the characteristics of the individuals providing the information for the study • Cannot reach people who do not have access to internet or do not have adequate skills of using internet or computers

16. Administering Surveys with Interviews • Overview • The interviewer asks questions and records the responses in a personal verbal interaction • Advantages • The interviewer and respondent often establish a rapport that can motivate the respondent to answer all the questions and complete survey (high response rate) • The interviewer is present to answer questions from the participant • Offers the researcher the flexibility to react to the respondent's situation, probe for more detail, seek more reflective replies, and ask questions which are complex or even personally intrusive

17. Administering Surveys with Interview • Overview (Cont.) • Disadvantages • High cost • It takes time to collect enough data for a complete survey, and time translates into payroll costs and sometimes payment for the participants • Interviewer bias • The interviewer may subtly bias the respondent’s answers by inadvertently showing approval or disapproval of certain answers • Interviewers may have expectations that could lead them to “see what they are looking for” in the respondent’s answers • Certain types of questions are not convenient to ask, particularly in phone interviews where the respondent does not have chance to look at the questionnaire • Types of interviews • Face-to-face interviews • Telephone interviews • Focus group interviews

18. Face-to-Face Interviews Require the interviewer and respondent to meet to conduct the interview Tend to be quite expensive and time-consuming Most likely to be used when the sample size is fairly small and there are clear benefits to a face-to-face interaction Telephone Interviews Used in interviews for large-scale surveys Less expensive than face-to-face interviews Computerized telephone survey technique (with a computer-assisted telephone interview system) lower labor and data analysis costs Administering Surveys with Interview

19. Administering Surveys with Interview • Focus Groups Interviews • A group of about 6 to 10 participants are brought together for a period of usually 2 to 3 hours • There is usually some sort of monetary or gift incentive to participate • Questions tend to be open-ended and asked of the whole group • Group interaction is possible • The interviewer must be skilled in working with the group both to facilitate communication and to deal with problems that may arise • The group discussions are usually recorded and then analyzed to find themes and areas of group consensus and disagreements • Researchers usually prefer to conduct at least two or three discussion groups (not many due to the high cost) on a given topic to make sure that that the information gathered is not unique to one group of people

20. Sampling • Why Sampling • Unless a survey population is very small, it is rarely to survey the entire relevant population • With proper sampling, researchers can make estimations or generalizations regarding the entire population • Sampling Techniques • Nonprobability sampling • The probability that any particular member of the population is chosen is unknown • Not as sophisticated as probability sampling, but quite common and useful in many circumstances, especially in experimental methods • Probability sampling • Each member of the population has a specifiable probability of being chosen • Must be used when it is important to accurately describe the population • Usually used in survey studies

21. Nonprobability Sampling • Haphazard Sampling (“Convenience” Sampling) • Can be described as “take-them-where-you-find-them” method of obtaining participants • e.g. Select a sample of students at WSU in any way that is convenient, such as asking your classmates or friends, visiting computer labs or libraries, etc. • Likely to introduce biases into the sample so that the sample may not be an accurate representation of the population of interest • e.g. Students studying in computer labs or libraries may be more devoted to learning • The experiment results may not generalize to the intended population but instead may describe only the biased sample obtained

22. Nonprobability Sampling • Purposive Sampling • The purpose is to obtain a sample of people who meet some predetermined criteria • e.g. You want to recruit participants who are under 30 years old and college students, then you can visit some fraternity and sorority houses at WSU • A good way to limit your sample to a certain group of people • Quota Sampling • Used to choose a sample that reflects the numerical composition of various subgroups in the population • e.g. You want to have a sample of students that includes 19% freshmen, 23% sophomores, 26% juniors, 22% seniors, and 10% graduate students, then a quota sampling would make sure you have these percentages • Haphazard sampling technique is used to collect data • The problem remains that no restrictions are placed on how individuals in the various subgroups are chosen

23. Probability Sampling • Simple Random Sampling • Every member of the population has an equal probability of being selected for the sample • e.g. Suppose you want to sample 500 students who are female students at WSU (about 10,000 in total), then each student has 500/10,000 probability of being selected. A list of all female students at WSU would be needed, from which 500 students would be chosen at random to form the sample • Stratified Random Sampling • The population is divided into subgroups (or strata), and random sampling techniques are then used to select sample members from each stratum • Built-in assurance that the sample will accurately reflect the numerical composition of the various subgroups • Particularly important when some subgroups represent very small percentages of the population • e.g. If African Americans make up 5% of a city of 100,000, a simple random sample of 100 people may not include any African American, whereas a stratified random sample would include five African Americans chosen randomly from the population

24. Entire Population Illustration of Stratified Sampling

25. Probability Sampling • Cluster Sampling • A sampling technique where the entire population is first divided into groups or clusters, and a random sample of these clusters is then selected (all members in the selected clusters are included in the sample) • Usages • Used when the researcher cannot get a complete list of the members of a population they wish to study but can get a complete list of groups or 'clusters' of the population • Used when a random sample would produce a list of subjects so widely scattered that surveying them would prove to be far too expensive Suppose you need to conduct personal interviews with freshmen college students in OH, if you take a simple random sample of all freshmen college students in OH, you may wind up with respondents who come from all over the state, which means that your interviewers are going to have a lot of traveling to do. In cluster sampling, you follow the steps: 1) Divide population into clusters (usually along geographic boundaries); 2) Randomly sample clusters; 3) Interview all people within sampled clusters

26. Entire Population Illustration of Cluster Sampling

27. Evaluate Samples • Representativeness of Samples • Samples should be representative of the population from which they are drawn • Randomly sample from a population that contains all individuals in the population • Contact and receive completed responses from all individuals selected to be in the sample • Sampling Frame • The actual population of individuals (or clusters) from which a random sample is drawn • Rarely perfectly coincides with the population of interest • e.g. If you define your population as residents of Dayton, the sampling frame may be a list of telephone numbers that you will use to contact residents between 5 and 9PM. This sampling frame excludes persons who do not have telephones or whose schedule prevents them from being at home when you are making calls

28. Evaluate Samples • Response Rate • The percentage of people in the sample who actually completed the survey • e.g. If you mail 1,000 questionnaires to a random sample of adults in your community and 500 are completed and returned to you, the response rate is 50% • Indicates how much bias there might be in the final sample of respondents • Non-respondents may differ from respondents in any number of ways (e.g. age, income, marital status, education, etc.) • The lower the response rate, the greater the likelihood that such biases may distort the findings and in turn limit the ability to generalize the findings to the population of interest • Ways to improve response rates • An explanatory postcard or letter can be sent a week or so prior to mailing the survey • Follow-up reminders and even second mailings of the questionnaire are often effective in increasing response rates • It often helps to have a personally stamped return envelope rather than a business reply envelope • An incentive may be necessary to increase response rates (e.g. cash, a gift, or a gift certificate)