Surveys, interviews, and focus groups

1. Surveys, interviews, and focus groups Lorrie Faith Cranor September 2011

2. Surveys, interviews, and focus groups • Surveys • Ask people set list of questions (possibly with conditional questions or branching) with multiple choice or free response answers • Can be conducted in person, by postal mail, by phone, online • Interviews • Ask people open-ended questions, with follow-up questions depending on their responses • Usually conducted in person or by telephone • Focus groups • Group interview, usually with 5-10 participants • Usually conducted in person

3. What method should I use? • Survey • Allows you to collect data from a lot of people relatively quickly • Easy to collect data in standardized format, ready for analysis • May be difficult to answer “why” questions • Useful for validating hypotheses from smaller studies with a larger population • Interview • Allows you to probe mental models, what people think and why • Semi-structured interviews allow you to explore issues as they are raised • Allows you to clarify if people don’t understand a question • Focus group • Similar to interviews, but more efficient as you can interview 5-10 people at once • Less detail from any interviewee than you would get in an individual interview • Not great for testing usability because participants probably won’t all use the software • Sometimes an opinionated individual can dominate a focus group • Hard to publish paper based only on one or two focus groups

4. Surveys

5. Designing good survey questions • Word questions clearly, without jargon or undefined abbreviations • Avoid leading questions, ambiguous terms, or emotionally-loaded terms • Design questions to evoke truthful responses • Non-threatening, don’t bias participants to provide what they think you want, protect confidentiality • Probe one dimension at a time • Design questions such that respondents are likely to provide a range of answers

6. Multiple-choice answers • Make sure answer choices are clear, mutually exclusive, cover entire space of possible answers • Make sure answer choices are at appropriate level of granularity • Where appropriate, allow respondents to indicate they don’t know, don’t have an opinion, or the question is not applicable to them • Use consistent rating scales throughout your survey • Be aware that the rating scale can skew responses (people like to think they are normal)

7. More survey design tips • Cluster similar questions together • Do not ask respondents to perform cognitively difficult tasks (unless you are testing their ability to perform these tasks) such as ranking more than 5 items • Use a clear and attractive layout • Pilot, pilot, pilot!

8. Interviews

9. Developing an interview script • Keep your questions fairly open-ended • You can follow-up with specific probes (“What files do you have on your computer that you consider valuable?” Follow up with “Do you have valuable photos? Videos? School work? Letters?”) • Start with general questions and get more specific so you get their unbiased impressions before you direct their thinking to particular details • A semi-structured approach allows you to adjust as needed • Write-out all your questions and follow-up probes

10. Role play and hypothetical scenarios • Appropriate for some interview studies • Give participant a role to play or put them in a hypothetical scenario • Imagine you just saw this message on your computer screen…. • Imagine your friend called you and told you he saw this message and asked you what to do….

11. Preparing for an interview • The day before the interview: • Print out: • Protocol – including detailed interview script • Official consent form • Payment sheet • Prepare: • Compensation payment • Audio/video recording (devices, batteries, extension cords, etc.)‏ • Additional material • Send an email to the interviewee to remind him/her of the date/time/place

12. During the interview • Explain the purpose of the study (unless you need to hide that)‏ • Ask to read and sign the consent form • If recording/video taping, turn it on! • Perform the study! • Debrief the person (if applicable)‏ • Ask to sign the signing sheet • Give the payment • Thank the person!

13. Coding interview data • Transcribe interview data (yourself or with hired transcription service) • Depending on purpose of study, transcribing only selected quotes may be sufficient • Iteratively review transcripts and create code for concepts mentioned by participants • As new codes are added, check to see whether those concepts were mentioned in previously analyzed transcripts • Keep track of how many participants mentioned each concept to find concepts that resonate with a lot of participants • Group similar codes together into categories • Note interesting quotes

14. Analyzing interview data • Interview can be formally analyzed through a coding process • Qualitative approach • Use concepts and categories to develop theory (Grounded theory approach, does not start with hypotheses) • Quantitative approach • With large number of interviewees (~30), and questions that ask participants to provide numerical ratings, quantitative analysis may be feasible • Important to validate for coder reliability • Can be used to develop mental models

15. Focus groups

16. Planning a focus group • Develop very detailed script to guide you • Pre-screen participants to get a good mix of people who meet your criteria • Setup audio and video recorders, but don’t make people feel under surveillance • Helpful to have at least 2 people, a moderator and a note taker • Give people name tags with their first name only • Plan to do multiple focus groups to mitigate effects of dominant participant steering conversation

17. Conducting a focus group • Make the session fun, informal, relaxed feel • Provide drinks and snacks • Promote a free flowing conversation that engages all participants • Ask open ended questions • Show people multiple things and ask them to compare • Give demos or show videos to start-off discussion • Give people handouts and ask them to circle things they like/don’t like, or jot down first impressions before discussing with the group

18. Using videos in focus groups • Videos can provide concrete scenarios for people to discuss