TURBULENT TIMES FOR SURVEY METHODOLOGY

TURBULENT TIMES FOR SURVEY METHODOLOGY Damon Burton University of Idaho

WHAT DO YOU KNOW ABOUT SURVEY RESEARCH? • Jot down at least 3 types of surveys you’ve taken in the past few years. • What value or benefit do surveys have in your life? How do surveys help make your life better? • How much does survey data influence your life? • provide 1-2 examples

CHANGES IN SURVEY RESEARCH OVER 50 YEARS

SURVEYING THROUGH THE 1960’S Until the late 1960’s, most survey research was conducted via in-person interviews. Random sampling was developed in the 1940’s & 1950’s. Efficient surveying required sampling in stages, and clustering locations to reduce costs. Planning and conducting studies took months and sometimes years, with data collection alone normally taking months. Sample surveying was limited to only the most important topics.

SURVEYING THROUGH THE 1960’S Polite, well-dressed women interviewers came to a person’s home to conduct the interview. Icebreaker questions (e.g., How is your family?) were used to develop report. Responses were recorded via paper and pencil in great detail. Interviewers attempted to get people to respond thoroughly to each question. Every effort was made to show that they were genuinely interested in the respondent as a person.

SURVEYING THROUGH THE 1960’S Survey research took a major blow when the Literary Digest’s predicted the wrong winner in the 1936 Presidential Election. Based on a 20% response rate to over 10 million straw ballots mailed to 36% of U.S. households, Alf Landon was predicted to beat Franklin Roosevelt 55% to 41%. Roosevelt actually beat Alf Landon 61% to 37% for the most famous botched prediction in the history of American politics, thus discrediting survey research.

SURVEYING THROUGH THE 1960’S Phone surveys were not practical because only one-third of households had phones in 1936. Area codes didn’t exist so operators had to place calls, making them extremely costly. Call quality was low, and many households shared party lines with other households. Long distance was used mostly for emergencies. Sharing personal information over the phone to someone you didn’t know was unappealing.

SURVEYING THROUGH THE 1960’S Mail surveys were used extensively in the 1940’s for specialized populations. Adequate household lists did not exist for doing general public surveys, and response rates were low. Surveys were constructed with typewriters, limiting design and composition possibilities. Copies were duplicated on mimeo machines, with time demands high and quality low. The process was time-consuming, costly and often ineffective.

SURVEY RESEARCH IN THE 1970’S AND 1980’S Development of modern telephone and mail survey methods began in the early 1970’s. Area codes were developed, making direct dialing possible, and development of Wide Area Telecommunication/Telephone Service (WATTS) lowered long distance costs. Phone quality improve greatly, allowing conversations in a normal tone of voice. By 1970, 87% of U.S. households had phones. Random-digit-dialing (RDD) was also developed to systematically sample households.

SURVEY RESEARCH IN THE 1970’S AND 1980’S Research showed phone & in-person interviews yielded similar response rates and results. Phone calls came without prior notice, normally in the early evening hours. Interviewers read a script in a conversational manner, and persons had only a few seconds to decide whether to respond. Scripted questions were read and respondents had to answer in predesigned ways. Open-ended questions were eliminated and elaboration discouraged to reduce call length.

SURVEY RESEARCH IN THE 1970’S AND 1980’S Breakthroughs such as photo copy machines made mail surveys more efficient. Personal computers and better printing methods streamline construction of surveys. Survey researchers personalized mailings and used multiple followups to raise response rates. Costs came down for conducting surveys, allowing smaller organizations in any area of the country to collect survey data. Most government surveys were done by mail, and public surveys were conducted by phone.

SURVEY RESEARCH IN THE 1990’S AND 2000’S Gated communities and locked apartment buildings reduced access to people’s homes. Unlisted phone numbers and removal of addresses from phone books reduced sample inclusiveness. Telemarketers used survey methods to market products over the phone, increasing the number of unsolicited calls and raising public mistrust. The public responded with unlisted numbers, answering machines, caller ID and call blocking. Respondents said “no” to almost all requests.

SURVEY RESEARCH IN THE 1990’S AND 2000’S Cell phones made phones personal not household devices. Scientific sampling became more difficult because not all persons had an equal or known chance of selection. Getting a list of cell numbers is also a problem. Conversations while people were driving and dropped calls are other cell problems. The result is a much lower proportion of people who can be reached by phone and who will agree to be surveyed.

SURVEY RESEARCH IN THE 1990’S AND 2000’S Computer technology has changed survey methodology significantly. Computer-Assisted Telephone Interviewing (CATI) allowed scripted responses to be entered directly into the computer. Touchtone Data Entry (TDE) had respondents providing responses to the computer by pressing keys on their touchtone phone. Interactive Voice Response (IVR) used the computer to conduct an interview and register vocal as well as keyed respondent answers.

SURVEY RESEARCH IN THE 1990’S AND 2000’S Internet surveying changed survey research. Cost savings were huge because interviewer wages, long-distance charges, postage, printing and data entry costs were eliminated. Web surveys also made finding a sample easier and quicker, and increased sampling locales. Programs such as Survey Monkey made constructing surveys simpler and better because the survey process was engineered. Conducting survey research became easier and quicker on-line.

SURVEY RESEARCH IN THE 1990’S AND 2000’S Internet surveying also presents a host of challenges and problems. Major gaps exist in computer and internet access and computer skills for certain samples. Lack of standards for creating email addresses that would allow development of a sampling algorithm has been problematic. Internet surveys are limited to populations with high internet access and skill levels such as certain professional organizations and purchasers of computer-related products.

SURVEY RESEARCH IN THE 1990’S AND 2000’S The Internet has changed our culture. Over time, distrust of the Internet has grown because of cyber crimes (e.g., identity theft) and viruses. Many people now avoid responding to unsolicited emails, particularly ones that involve clicking on a link. Email and texting have become preferred methods for communicating at work and home. Survey researchers must promote the value of their surveys from other communications.

SURVEY RESEARCH IN THE 1990’S AND 2000’S The 18 to 30-year-old population segment is the most difficult to get to respond to surveys. Mail surveys can now achieve higher response rates than telephone surveys, although identifying a sample has become harder. Many survey researchers use multiple survey modes (e.g., phone, mail, internet). Respondents now can be given a choice of survey mode. Refusal to respond via one mode often prompts requests to respond using another mode.

GETTING BEYOND THE TURBULENCE Certain survey modes are better for specific populations, topics and sponsors than others. Survey research has shifted from one to several survey modes for the same data collection. In the mixed-mode era, questions must work across modes, and modes must be complementary to maximize response rates. Goal is to design scientifically-sound data collection systems that allow precise estimates of behaviors and attitudes of a population by sampling only a small proportion of them.

FROM TOTAL DESIGN TO TAILORED DESIGN Dillman’s (1978) 1st edition focused on the Total Deign Method (TDM-1) that personalized all aspects of surveys. TDM-1 used the same procedures for all populations and situations (i.e., one size fits all). The Tailored Design Method (TDM-2) tailors surveys to fit specific populations & situations. In TDM-2, criteria for successful surveying were expanded to include coverage, sampling, measurement and nonresponse error. Layout design and mixed modes were added.

3RD EDITION OF THE TAILORED DESIGN METHOD Huge changes have occurred in the past 9 years as technology has confronted culture. The Internet has reached most businesses but is still unavailable in many homes. Phones have become personal rather than household tools and become harder to access. Survey responses are more dependent on respondent choice. Survey design, mode and choice require greater up-front preparation and understanding of how to best reach the “target population.”

3RD EDITION OF THE TAILORED DESIGN METHOD Consideration of internet applications influences coverage, sampling, question construction and choice of modes to achieve high response rates. Many surveys demand mixed modes to best target a specific population and/or topic. Visual design concepts can enhance effectiveness to create a more “personal feel.” Longitudinal and internet panel surveys are particularly sensitive to visual design elements. Sponsorship of surveys may also dramatically impact their design and implementation needs.

The End

TURBULENT TIMES FOR SURVEY METHODOLOGY