So Now You Want to Do a Survey...

1. So Now You Want to Do a Survey... Words of Advice, Words of Caution Jay P. Paul, PhD UCSF Center for AIDS Prevention Studies

2. Key Questions To Consider When Asking Questions • Is this item understandable to the participant? • Is this item something the participant can answer? • Is this item something the participant will be willing to answer honestly? • How important is this item compared to others I have to avoid respondent burden?

3. Areas to Consider in Designing Your Survey • What type of survey methods are out there and what can they do for me? • What are the things to consider in maximizing the quality of data with respect to questions I ask? • How do I consider using existing scales and putting it all together?

4. Modes of Survey Delivery

5. Survey Method Options • IAQ • SAQ • Including Diary Methods, Mail Surveys • CASI/CAPI (and A-CASI) • CATI • T-ACASI • Internet

6. Interviewer-Administered Questionnaire (IAQ) • Most flexible and responsive to respondent re: comprehension of question, selection of response options. • Able to handle skip patterns that could confuse the respondent in a self-administered questionnaire. • Human element, rapport & responsiveness also can introduce problems in lack of standardization of delivery, self-presentation.

7. Self-Administered Questionnaire (SAQ) • Allows for greater privacy than face-to-face interviewing • Tends to be associated with higher rates of reporting of sensitive behaviors • Literacy requirements for respondent • Cannot be adjusted to be responsive to needs of individual participant • Cannot handle complex “skip patterns” • Low-cost survey delivery option

8. Diary Methods • Provides optimal potential for recording of accurate rates of target behaviors • Recall • Exploration of variations over time • Allows for question design to examine co-occurrence of key behaviors (e.g., sexual behavior and drug use) • Burden of consistent completion of information means that data may be missing in possibly non-random pattern, or completed at a later time than designed. • NOTE: Can be useful to design for Web-based implementation.

9. Mail Surveys • Many of considerations of SAQ are applicable -- with format even more crucial. • Many respondents report liking the flexibility of completing surveys at their own pace. • Need for clear and easy format to maintain respondent interest and ease of completion. • Critical to have contact information and clear procedures to follow up on respondents to ensure adequate response rates.

10. Computer-Assisted Self-Interview /Personal Interview (CASI/CAPI) • Survey is delivered via computer (can be laptops for maximal flexibility), with respondent keying in responses. • Usual mode is “Audio-CASI” or A-CASI, which involves respondent listening to audio recording of survey as they see items on-screen. • For special PCs with touch screens, can avoid keyboard and use fingers to select choice by touching on-screen.

11. CASI/A-CASI Advantages • Privacy • Branching in complex questionnaires • Automated consistency checks • Automated range checks • Automated adaptation of question wording based upon prior responses • Audio-CASI allows for standardized verbal delivery of all questions and does not require respondent literacy

12. Computer-Assisted Telephone Interview (CATI) • List samples of “live” residential phone lines are called to screen households and determine eligibility of members. • Once consent is obtained, interviewer uses a computer to guide them through asking survey questions, providing prompts, entering data, etc. as in A-CASI. • Interviewer also has some flexibility in dealing with respondent comprehension/literacy level.

13. CATI Advantages • Opportunity to access representative sample (costly with eligibility requirements that are less common). • Some advantages of live interviewer, but also anonymity of phone call. • Has capacities of computer-assisted technology with respect to branching, adaptation of wording, consistency/range checks, etc.

14. Telephone Audio Computer-Assisted Self-Interview (T-ACASI) • First a telephone interviewer contacts a household, screens for an eligible respondent, and secures consent. • Phone call then transferred to automated system where computer-controlled, pre-recorded questions are read aloud. • Respondent provides answers by pressing keys on a touch-tone phone. • At end, respondent can be returned to live person to close out interview.

15. Internet Surveys • Access to large sample at relatively low cost in relatively short time frame. • May provide access to difficult-to-reach populations. • As with other computer-assisted methods, may result in higher rates of reporting of sensitive information, and provides consistency and validity checks. • Rates of internet access are growing (if still disparities exist in different populations). • Participation rates of target population hard to gauge • Dropout rates • Direct data entry into database facilitates cleaning and analysis.

16. Summary: Survey Delivery Considerations • Mode of survey delivery clearly influence: • Respondent comfort and likelihood of self-disclosure (due to level of privacy) • Respondent ease of navigating through the survey if there are any skip patterns • Time and effort expended per respondent • The audience you can reach with your survey • Quality of data, data entry and analysis • While important, how you actually ask the questions is just as critical -- our next focus.

17. Survey Construction

18. Developing the Survey: Question Organization • Topics should be generally organized from least to most intimate • Initial questions are crucial -- respondent must get “hooked” by at least some questions which are felt to be personally salient • Initial questions must also require low effort (cognitive demands) to answer • Thus, if possible, avoid demographic questions to start (especially income!)

19. Developing the Survey: Question Organization • A survey -- no matter how delivered -- is a type of conversation. • Topics should follow an orderly sequence, with common questions grouped together. • This flow allows transitions to more intimate matters without jarring or acute awareness. • Transitional phrases can help in shifting set. • All this also provides respondent with notion that there is a rationale to questions asked.

20. Question Ordering • Be careful with filter questions: if not worded carefully, respondents will learn that a “no” response means avoiding subsequent questions. • Be careful about summary attitudinal questions: more specific queries to begin will influence summary responses by forcing respondent to think about their overall beliefs more carefully.

21. Question Ordering • Having similar response choices reduces cognitive burden on respondent; grouping items with such options together is thus useful. It can also be useful to have the same number of options to reduce complexity (e.g., always using a four-point or five-point scale). • However, arrange types of questions to provide some variety and reduce response set. • Avoid asking unnecessary questions!!

22. Question Wording/Construction

23. Comprehension/Interpretation • Not just question of literacy, but influenced by cognitive burden and attention required • Wording choices • Length and complexity of survey items • Ordering of clauses • Consistency of phrasing • Avoid “not’s” or items whose meaning will be lost by inattention to a single word • Phrasing also influences response distribution along a Likert-type agree-disagree scale

24. Cultural Considerations • Individuals can come up with very different understandings of survey items, based upon their personal experiences. • The meanings that respondents bring to the survey experience can be strongly influenced by culture, acculturation and other background variables -- not just education and economic status. • These differences are highlighted in the vast literature that exists on different response patterns of different ethnic groups to the same standardized scales. (For a review of some issues, see Pasick, Stewart, Bird & D’Onofrio, 2001.)

25. Key Considerations • Examine all survey items with eye for their appropriateness for target population. • Be aware of pre-existing measures so as to not have to reinvent the wheel. • When using pre-existing scales, review in same way for clarity & ease of response. • Get feedback on item wording and construction from as many colleagues as possible. • Pre-test survey, using methods first described by Cannell et al. (1989). Cannell C, Oksenberg I, Kalton G, Bischoping K & Fowler F. (1989). New Techniques in Pretesting Survey Questions (NCHSR #HS 05616). Survey Research Center, University of Michigan.

26. Cognitive Burden & Recall Demand • Recall accuracy is influenced by a variety of factors, including: • salience/vividness of behavior, • time frame, • complexity of response information necessary (ever, counts, close-ended categories, etc.), • frequency and consistency of behavior, • effort respondent is willing to exert, • cueing techniques and prompts

27. Time Frame - Sexual/Drug Use Behaviors • 3 month time frame generally considered the outside range of accurate reporting; some suggest limiting to 1 month. • Longer time frame may be possible if asking if a behavior ever happened. • One important timeframe consideration is dealing with infrequent behaviors -- what do you want to capture with respect to either behavior events or possible behavior change (if longitudinal study)?

28. Sensitivity • Item sensitivity not consistent across populations (e.g., sexual behavior among MSM vs. general population, drug use behavior among heavy drug abusers/IDUs, discussing sexuality among different ethnic groups). • Rationale for sensitive questions helps response rates. • Asking if “ever” occurred first, prior to asking about (for example) the last month, may be less charged for respondent. • Importance of neutral wording of question and (if IAQ) careful training of interviewer

29. In Closing ... • Selecting your methods of data collection must be carefully considered based upon target population, study questions and design, and costs. • No matter what the method of data collection, your instrument is critical. • Survey design is a combination of art and science, with the “art” strongly reliant upon interviewing skills. • The science of survey design currently has a wealth of research literature to examine and guide you.