Asking Users and Experts

1. Asking Users and Experts Bobby Kotzev Adrian Sugandhi

2. Outline • Asking Users • Interviews • Questioners • Asking Experts • Inspections • Walkthroughs

3. Interviews, Survey, or Focus Group? • Interview • Take significant time and need more resources • Survey • Large data set • Focus group • Seeking multiple point of views in a shorter period of time

4. Preparing for Interviews:Things to be aware • Be alert to unconscious biases • Honesty • Social desirability • Prestige response bias • Sensitive or highly personal topics – survey? • Consider the interviewee as he/she is “Doing you a favor”

5. Preparing Interviews:Outcome Analyses • Outcome Analyses • Plan outcome-based interviews • Associate questions with the goals • Capture desired outcomes • Differentiate between outcome and solution • Organize the outcomes • Rate the outcomes for importance and satisfactions • Importance + (Importance-Satisfaction) = Opportunity • Use the outcomes to jump-start innovation

6. Preparing Interviews:Pre-interview • Identify the objective of the study • Select type of interviews • Medium of interview (in person/phone) • Decide how you will analyze the data • Write the questions • Avoid long questions (Brevity) • Avoid compound sentences • Avoid using jargon • Avoid leading questions • Avoid Biases • Clarity • Avoid future prediction questions • Avoid inaccessible topics • Test your questions

7. Preparing Interviews:Players in the Activity • Participants • 6-10 of each user type (diversity) • Interviewers • Ensure participants understand the questions • Must be skilled • The note-taker • Interviewers can focus more on body language • The videographer • Whenever possible, record the interview session

8. Conducting Interviews: Sections • Introduction • 5-10 Minutes • Warm-up • 5-10 Minutes • Main section • 85-100 Minutes • Cool off period • 5 Minutes • Closing

9. Preparing for Interviews: Additional Preparations • Run pilot study • Be professional • Dress similarly and appropriately • Prepare informed consent • Check and familiarize yourself with recording equipment • Record answers exactly

10. Interviews Types • Interview Types • Open-ended • Unstructured • Structured • Semi-structured

11. Unstructured Interviews • A conversation that focus on a particular topic • Open questions • Can generate rich data which the interviewer haven’t thought about • Generates a lot of unstructured data • Impossible to replicate • Hard to analyze

12. Unstructured Interviews: Prepare • Have an agenda • Be prepared to follow new lines • Pay attention to ethical details • Respond with sympathy but make sure to avoid putting ideas in the user’s head • Analyze data as soon as possible after the interview

13. Structured Interviews • Predetermined questions, similar to a questionnaire.

14. Semi-Structured Interviews • Starts as a structured interview but can change direction and inject open-ended questions depending on responses • Avoid preempting answers • Probes – “Do you want to tell me anything else about…”

15. Interviewer’s Role • Do not interrupt • Keep on track • Unstructured interview easy goes off-track • Silence is golden • “permission” to provide more detail • Remain attentive • Asking the tough questions • Wait until you develop rapport • Using examples • Watch for generalities • Do not force choices

16. Interviewer’s Role (cont’d) • Watch for markers • Key events to probe for more rich information • Select the right type of probes • Know when to move on • Reflecting • Summarize, reword, or reflect responses • Empathy and antagonism • Transitions • Transition smoothly from one topic to another

17. Monitoring the Relationships with the Interviewee/Participant • Watch participant’s body language • Nervous, tenses – go to easier question or restate purpose/motivation of the study • Fighting for control • Ask yourself why one refusing your questions • Hold your opinions • Dos and Don’ts

18. Interview: Data Analysis and Interpretation • Must be analyzed shortly after each interview • Categorizing • Affinity diagram • Qualitative analysis tools • Look for patterns

19. Interview: Communicate the Findings • Over time • By topic • By participant • Vehicles for communicating the results • Summarized poster • Identify follow-up activities based on the results • Table of recommendations (uncovered issues and next steps)

20. Questionnaires • Good, established technique for collecting demographic data and users opinion • Can be Closed and Open. • Can be distributed to large number of people

21. Questionnaires: Designing • Start with basic demographic data • Make questions clear and specific • When possible ask closed questions and offer a range of answers • Include no-opinion option • Ordering matters • Avoid complex and compound questions • Use intuitive, consistent scaling • Avoid jargons • Provide clear instructions how to complete the questioner

22. Questionnaires: Question Types • Checkboxes • Likert Scales • Semantic differential scales Gender: Maleq Femaleq

23. Questionnaires: Administering • Well designed • Include stamped self addressed envelope • Provide a short version • Contact and respondents trough a follow-up, mail, e-mail phone • Offer incentives like payments

24. Questionnaires: Online Forms • Pros: • Reach large audience • Response is fast • No postage costs • Data is already in a electronic format and can be imported easily in DB • Errors can be corrected easily and fast • Cons: • Obtaining random sample is difficult – web users represent only a certain demographic. • Participants are self selecting – proclaimed nonscientific

25. Questionnaires: Online Forms (cont’) • Construct Online questioners based on their paper forms • Run pilot studies

26. Asking Experts • Experts will be power users but also could be involved with the development of similar, successful products so they have an deeper understanding.

27. Asking Experts: Inspections • Heuristic evaluations • evaluate user-interface against a predetermined guidelines and principles • Visibility of system status • Are users kept informed about what is going on? • Is appropriate feedback provided within reasonable time about a user action? • User Control and freedom • Are there ways of allowing users to easily escape from places they unexpectedly find themselves in? • Consistency and standards • Are the ways of performing similar actions consistent?

28. Asking Experts: Inspections (cont’) • Different heuristics are needed for different products, devices and software • Using heuristics and experts can reveal issues fast and inexpensively – 5 experts can reveal 75% of issues - graph • Performing heuristic evaluation • Spend 1-2 hours with the product. • At least two passes – one to get the feel, second to walk trough the interface and address all aspects of usability. • Have a specific task in mind when evaluating functional products

29. Asking Experts: Inspections (cont’) • Heuristic evaluation of websites • Internal consistency • Simple dialog • Shortcuts • Minimizing the user memory load • Preventing errors • Feedback • Internal locus of control • Layout • Internal consistency

30. Asking Experts: Problems • Reporting issues which are not there – (Bill Bailey 2001)

31. Asking Experts: Walkthroughs • Walk through a task and to notice problems • “Cognitive walkthroughs” – simulating the user problem solving process for a task • Figure if the users will know what to do, how to do it and weather the action was correct or not. • Record assumptions, side issues and summaries of results • Pluralistic walkthroughs • Users developers and usability experts get together to step through a scenario

32. Summary • Interviews • Fresh input • Questionnaires • Reach the masses • Experts • Easy way to discover most problems