Collecting Data from Users

1. Collecting Data from Users

2. Uses • User and task analysis • Prototype testing • On-going evaluation and re-design

3. Data collection methods • Questionnaires • Interviews • Focus groups

4. Definitions • Survey: • (n): A gathering of a sample of data or opinions considered to be representative of a whole. • (v): To conduct a statistical survey on. • Questionnaire: (n) A form containing a set of questions, especially one addressed to a statistically significant number of subjects as a way of gathering information for a survey. • Interview • (n): A conversation, such as one conducted by a reporter, in which facts or statements are elicited from another. • (v) To obtain an interview from. • American Heritage Dictionary

5. Surveying • Sample selection • Questionnaire construction • Data collection • Data analysis

6. Surveys – detailed steps • determine purpose, information needed • identify target audience(s) • Select method of administration • design sampling method • design prelim questionnaire • including analysis • Often based on unstructured or semi-structured interviews with people like your respondents • pretest • Revise, pretest… • administer • analyze

7. Why survey as method? • Answers from many people, including those at a distance • Relatively easy to administer, analyze • Can continue for a long time

8. Surveys can collect data on: • Facts • Characteristics of respondents • Self-reported behavior • This instance • Generally/usually • Opinions and attitudes: • Preferences, opinions, satisfaction, concerns

9. Some Limits of Surveys • Reaching users is easier than non-users • Relies on voluntary cooperation, possibly biasing the sample • Questions have to be unambiguous, amenable to short answers • You only get answers to the questions you ask; you generally don’t get explanations • The longer or more complex the survey the less cooperation

10. Some sources of error • Sample • Question choice • Can respondents answer? • Question wording • Method of administration • Inferences from the data • Users’ interests in influencing results • “vote and view the results” CNN quick vote: http://www.cnn.com/

11. When to do interviews? • Need details that can’t get from survey • Need more open-ended discussions with users • Small #s OK • Can identify and gain cooperation from target group • Sometimes: want to influence respondents as well as get info from them

12. Sample selection

13. Targeting respondents • About whom do you want information? • About whom can you get information? • E.g. non-users are hard to reach

14. Sampling terminology • Element: the unit about which info is collected; basis of analysis. E.g., “User” • Universe: hypothetical aggregation of all elements. “All users” • Population: a specified aggregation of survey elements. “People who have used this service at least once in the last year.” • Survey population: aggregate of elements from which the sample is selected.“People who use the service at least once during the survey period.”

15. Terminology, cont. • Sampling unit: elements considered for selection. • Sampling frame: list of sampling units. • Observation unit: unit about which data is collected. Often the same as unit of analysis but not always. E.g. one person (observational unit) may be asked about the household (unit of analysis). • Variable: a set of mutually exclusive characteristics such as sex, age, employment status. • Parameter: summary description of a given variable in a population. • Statistics: summary description of a given variable in a sample.

16. Sample design • Probability samples • random • stratified random • cluster • Systematic • Size: if 10/90% split, 100; if 50/50, 400; • Yale’s slide • 30-50 in each cell • GOAL: Representative sample • Non-probability sampling • convenience sampling • purposive sampling • quota sampling

17. Representative samples • Which characteristics matter? • Want the sample to be roughly proportional to the population in terms of groups that matter • E.g., students by gender and grad/undergrad status: • http://opa.vcbf.berkeley.edu/IC/Campus.Stats/CampStats_F00/CS.F00.Table.F2.htmgraduate/undergraduate…a

18. Active vs passive sampling • active: solicit respondents • Send out email • Phone • Otherwise reach out to them • Follow up on non-respondents if possible • passive – e.g. on web site • Response rate may be unmeasurable • heavy users may be over-represented • Disgruntled and/or happy users over-represented

19. Response Rates • low rates may > bias • Whom did you miss? Why? • How much is enough? • Babbie: 50% is adequate; 70% is very good • May help if they understand purpose • Don’t underestimate altruism • Incentives may increase response • Reporting back to respondents as a way of getting response

20. Example of a careful sample design • http://www.pewinternet.org/reports/reports.asp?Report=55&Section=ReportLevel1&Field=Level1ID&ID=248

21. RECAP • We collect data from users/potential users for: • User and task analysis • Prototype testing • On-going evaluation and re-design • Methods include: • Surveys, interviews, focus groups • Different methods useful for different purposes

22. Some Issues Common to Different Methods • Know your purpose! • Match method to purpose and feasibility • Whom do you need/can you get to participate? • Population, sample composition, sample selection methods, size • Response rate, respondent characteristics (bias)

23. Common Issues, cont. • What do you need to know? • What can/will respondents tell you? • How will it help? • How do you ask what you want to know? • Question construction, wording • Question ordering • What do you do with results? • Reporting and analysis

24. Types of web surveys • Comprehensive • Quick polls – focused, one or few questions • http://www.gomez.com/ratings/index.cfm?topcat_id=19&firm_id=1768&CFID=295681&CFTOKEN=8136111 • Short, focused surveys • Guestbooks, user registration, user feedback • http://www.bookfinder.com/interact/comments/

25. Uses of surveys of web sites • Identify users • Describe their characteristics • Describe their behavior • Ask their needs, preferences • Assess user satisfaction/response • Identify user problems, dissatisfactions • Solicit ideas for improvement

26. Problems with Web Surveys • Population? • Size • Characteristics • Response rate? • Multiple responses from same person OK? • Biased sample? • Users competent to answer? • E.g., will they answer after they have used the site enough to be able to judge? • Non-users not represented; infrequent users under-represented?

27. Questionnaire construction

28. Questionnaire construction • Content • Goals of study: What do you need to know? • What can people tell you? • Conceptualization • Operationalization – e.g., how do you define “user”? • Question design • Question ordering

29. Topics addressed by surveys • Respondent (user) characteristics • Respondent behavior • Respondent opinions, perceptions, preferences, evaluations

30. Respondent characteristics • Demographics • General http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • Professional role (manager, student..) • User role (e.g., buyer, browser…) • Expertise – hard to ask • Domain • technology • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html • System

31. Behavior • Tasks (e.g., for what are they using this site?) • Site usage, activity • Frequency; common functions – hard to answer accurately • Self-reports vs observations • Web and internet use: Pew study

32. Opinions, preferences, concerns • content • Organization, architecture • interface • perceived needs • concerns • http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/privacy.html • Success, satisfaction • Subdivided by part of site, task, purpose… • other requirements • Suggestions

33. Problems with questions • Social acceptability • Privacy • “Do you use the internet to view pornography?” • Difficulty wording unambiguously • International concerns • Language • Social acceptability

34. Problems: topics hard to conceptualize, operationalize • e.g., “Why did you use the CDL today?” • Teaching, research… • my category of work or task • to find an electronic journal, locate a book, find a citation… • To locate kind of resource • to find material on x topic • the subject area • to save me time • i.e. I used this rather than another way of accessing same resource

35. Question construction

36. Question formulation • match respondents’ language • match respondents’ behavior • what do they want to tell you? • What can they tell you? • Recent CNN.com poll: “do you think the sentence given to x was too long, about right, too short.” What was the sentence? (What was the crime?) • Beware of compound questions, hidden assumptions: • ‘did you order something? If so, did you….’ – what if only browsing?

37. Question formulation: simplicity and clarity Complete the following sentence in the way that comes closest to your own views: 'Since getting on the Internet, I have ...' • ... become MORE connected with people like me. • ... become LESS connected with people like me. • ... become EQUALLY connected with people like me. • ... Don't know/No answer. http://www.gvu.gatech.edu/user_surveys/survey-1998-10/questions/general.html

38. Survey questions – format • open-ended http://www.bookfinder.com/interact/comments/ • “What is your job title?” _______ • “What do you find most difficult about your job?” • closed-ended (one answer; multiple answers) • http://www.useit.com/papers/surveys.html “Select the range that best represents the total number of staff…” 1-2 3-5 6-10… • paired characteristics/semantic differential Friendly__|___|___|___|____| Unfriendly

39. Question format, cont. • Ordinal scale/ “Likert scale” Strongly agree, agree, neutral, disagree, strongly disagree http://www.amazon.com/exec/obidos/tg/stores/detail/-/books/073571102X/rate-this-item/104-0765616-4703139 • Rating scale – matrix (usability survey, qn 5) • Always include neutral, “other,” “N/A” (not applicable)

40. Closed-ended questions • Answers need to be comprehensive and mutually exclusive; • OR • Allow people to give more than one answer • Tell them how to choose “Why did you come to SIMS?” • How many responses to allow? • As many as your respondent needs • Likert scale-like questions: • 5 OR 7 is usual; can respondent differentiate 7? • Very strongly agree; strongly agree; agree; neutral; disagree; strongly disagree; very strongly disagree? • Odd vs even: even allows neutral, odd forces a choices.

41. Filter and Contingency questions • 1. “Have you ever used our competitor?” • “If no, go to question 3.” • 2. “If yes, how would you rate….” • Did you apply to any graduate program other than SIMS? • If yes… • If no…

42. Layout: consistency Circle the answer that best matches your opinion. SIMS is a great place to study. Strongly agree Agree Neutral Disagree Strongly disagree UC Berkeley is a great university. Strongly agree Agree Neutral Disagree Strongly disagree IS214 is a great course. Strongly disagree Disagree Neutral Agree Strongly agree

43. Include Instructions! • One answer to each question, or multiples? • If online, can program to test and let user know they have violated guidelines. • Respond every time you visit this site? • Be sure it’s clear what to do if a question does not apply. • “Why did you choose SIMS over any other programs that accepted you?”

44. Question ordering • Group similar questions • Using headings to label parts of survey, topics • Funneling: • General to particular • Particular to general • Keep it short! Response falls off with length.

45. Pre-test, pre-test, pre-test! • With people like expected respondents • Looking for: • Ambiguous wording • Missing responses • Mismatch between your expectations and their reality • Any other difficulties respondents will have answering, or you will have interpreting their responses

46. Web Survey Problems • Who is the population? Self-selected sample. • Stuffing the ballot box • Cnn.com polls • How to know what response rate is • How to get responses (1) after they have used site (2) before they leave • What are you assessing and what are they responding to? • E.g., design of site, or content? Presentation of content, or content of content? • Loss of context – what exactly are you asking about, what are they responding to? • Are you reaching them at the appropriate point in their interaction with site?

47. Web Survey Problems II • Incomplete responses -- avoiding blank responses – and annoying users • Respondents may not know how long the survey is • Let people know • Multiple submissions

48. Survey administration

49. Human subjects considerations • not hurt by the process itself (e.g., uncomfortable questions) • not hurt by uses of the data • job etc not in jeopardy • not at risk for criminal prosecution • confidentiality or anonymity • Consent, voluntary participation • http://cphs.berkeley.edu

50. Ways of administering surveys • On a web site • Open access • Limited access • Email (with URL) • Phone • Paper • Mail • Users pick up on their own • Hand out in person • Fax • Face to face