Asking Users and Experts

1. Asking Users and Experts Changsung Moon Jaeyoung Lee

2. The Main Aims • Discuss when it is appropriate to use different types of interviews and questionnaires. • Teach you the basics of questionnaire design. • Describe how to do interviews, heuristic evaluation, and walkthroughs. • Describe how to collect, analyze, and present data. • Discuss the strengths and limitations of these techniques.

3. The Types of one-on-one Interviews • Unstructured (or open-ended) • Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order. • Ex) Suggest ways of collecting the interview data. • Structured • Most controlled type. • Interview may consist of closed-ended questions and interviewee must choose from the options provided. • Ex) Would you like to receive news from CNN? • Yes No Don’t know • Semi-structured • Combination of structured and unstructured • Ex) Would you listen to the news from CNN? • If yes, why? If no, why not?

4. The Types of one-on-one Interviews • Unstructured (or open-ended) • Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order. • Ex) Suggest ways of collecting the interview data. • Structured • Most controlled type. • Interview may consist of closed-ended questions and interviewee must choose from the options provided. • Ex) Would you like to receive news from CNN? • Yes No Don’t know • Semi-structured • Combination of structured and unstructured • Ex) Would you listen to the news from CNN? • If yes, why? If no, why not?

5. The Types of one-on-one Interviews • Unstructured (or open-ended) • Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order. • Ex) Suggest ways of collecting the interview data. • Structured • Most controlled type. • Interview may consist of closed-ended questions and interviewee must choose from the options provided. • Ex) Would you like to receive news from CNN? • Yes No Don’t know • Semi-structured • Combination of structured and unstructured • Ex) Would you listen to the news from CNN? • If yes, why? If no, why not?

6. The Types of one-on-one Interviews • Unstructured (or open-ended) • Questions or topics are open-ended, interviewee is free to answer in any manner, and topics do not have to be covered in any particular order. • Ex) Suggest ways of collecting the interview data. • Structured • Most controlled type. • Interview may consist of closed-ended questions and interviewee must choose from the options provided. • Ex) Would you like to receive news from CNN? • Yes No Don’t know • Semi-structured • Combination of structured and unstructured • Ex) Would you listen to the news from CNN? • If yes, why? If no, why not?

8. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

9. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

10. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

11. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

12. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

13. Guidelines for Developing interview questions (Things to avoid) • Long questions • They are difficult to remember. • Compound sentences • Ex) Wrong: “How do you like this cell phone compared with previous ones that you have owned?” • Right: “How do you like this cell phone?”, “Have you owned other cell phones?” If so, “How did you like it?” (easier to understand and record) • Using jargon • The interviewee may not understand. • Leading questions • Ex) Wrong: “Most of our users prefer the new look and feel of our site over the old one. How do you feel?” • Right: “How do you feel about the visual appearance of this website?” • Unconscious biases • Ex) Wrong: “Safety experts recommend using a travel agent instead of booking your travel online. Do you feel safe using travel websites?” • Right: “Do you feel that booking travel online is more or less confidential than booking through a travel agent?”

14. Timeline for an Interview

15. Group Interviews • ‘focus group’: one form of group interview • Normally 3 to 10 people are involved • Participants are selected to provide a representative sample of typical users; • they share certain characteristics • Benefit is that it allows diverse or sensitive issues that would otherwise be missed

16. Questionnaires • Questions can be closed or open (similar to interview) • Closed questions are easier to analyze • can be administered to large populations • Paper, email and the web used for dissemination • Electronic questionnaires • Data goes into a database and is easy to analyze • Online questionnaires • Sampling can be a problem when the size of a population is unknown

17. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

18. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

19. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

20. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

21. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

22. Advice for Designing a Questionnaire • Make questions clear and specific • When possible, ask closed questions and offer a range of answers • Consider including a “no-opinion” option • General questions should precede specific ones • Avoid complex multiple questions • The range of answers is appropriate and does not overlap • Ex) 15-20, 20-25 (X) 15-19, 20-25 (O) • The ordering of scales should be intuitive and consistent • Ex) 1 to 5, for 1 to indicate low agreement and 5 to indicate high agreement • Avoid jargon • Prove clear instructions on how to complete the questionnaire • Keep the questionnaire as compact as possible

23. Question and Response Format • Different types of questions require different types of responses. • Check boxes and Ranges • Ex) under 21 21-65 over 65 • Likert Scales • Ex) The use of color is excellent: (where 1 represents strongly agree and 5 represents strongly disagree) • 1 2 3 4 5 • Semantic differential scales • Ex) Exciting Boring

24. What to Do with the Data • Identify any trends or patterns • Using a spreadsheet like Excel can help in initial analysis • Often simple statistics are needed • Bar charts can be used to display data graphically • More advanced statistics can be used • show whether there is a relationship between question and responses

25. ASKING EXPERTS

26. Why Asking Experts? • Sometimes users are not easily accessible. • Involving users is too expensive • Evaluating a interface design takes so long time. • Skillful experts can capture many of the usability problems by themselves. • We can employ just few experts.

27. Two Different Technologies for Asking Experts • Inspections • Heuristic Evaluation • Walkthrough • Cognitive Walkthrough • Pluralistic Walkthrough

28. Heuristic Evaluation

29. Heuristic Evaluation • A discount method for quick, cheap, and easy evaluation of the user interface. • Developed by Jacob Nielsen in the early 1990s based on heuristics • distilled from an empirical analysis of 249 usability problems

30. Heuristic Evaluation -10 Heuristics • Visibility of system status • Match between system and the real world • User control and freedom • Consistency and standards • Help users recognize, diagnose, and recover from errors • Error prevention • Recognition rather than recall • Flexibility and efficiency of use • Aesthetic and minimalist design • Help and documentation “Are users kept informed about what is going on?” “Is appropriate feed back provided within reasonable time about a user’s action?”

31. Heuristic Evaluation • Revised for current technology • HOMERUN for the Web : These heuristics are more useful for evaluating commercial websites. • High-quality content • Often updated • Minimal download time • Ease of use • Relevant to users’ needs • Unique to the online medium • Netcentric corporate culture • Heuristics still needed for new technologies • mobile devices, wearable interfaces, virtual worlds, etc.

32. How many Experts? • Nielsen found that about 5 evaluations found 75% of the problems • Above that you get more, but at decreasing efficiency

33. Three Stages for Doing Heuristic Evaluation • Briefing session to tell experts what to do • Evaluation period of 1-2 hours in which: • Each expert works separately • Take one pass to get a feel for the product • Take a second pass to focus on specific features • Debriefing session in which experts work together to prioritize problems

34. Advantages and Problems • Best experts have knowledge of application domain & users • Few ethical & practical issues to consider • Can be difficult & expensive to find experts • Biggest problems • important problems may get missed • many trivial problems are often identified

35. Walkthroughs • Cognitive Walkthroughs • Pluralistic Walkthroughs

36. Cognitive Walkthroughs “Cognitive walkthroughs involve simulating a user’s problem-solving process at each step in the human-computer dialog, checking to see if the user’s goals and memory for actions can be assumed to lead to the next correct action.” (Nielsen and Mack, 1994)

37. Four Steps of Cognitive Walkthroughs • Analysis Tasks • Evaluate each task • Compile the record of critical information • Suggest revised design

38. Steps involved in Cognitive Walkthroughs – First Step • The characteristics of typical users are identified and documented. • Sample tasks are developed that focus on the aspects of the design to be evaluated. • A description or prototype of the interface is also produced, along with a clear sequence of the actions needed for the users to complete the task.

39. Steps involved in Cognitive Walkthroughs – Second Step • Designers and one or more expert evaluators then come together to do the analysis • The evaluators walk through the action sequences for each task, placing it within the context of a typical scenario.

40. Steps involved in Cognitive Walkthroughs – Second Step • While experts are evaluating each task, they try to answer the following questions • Will the correct action be sufficiently evident to the user? • Will the user notice that the correct action is available? • Will the user associate and interpret the response from the action correctly?

41. Steps involved in Cognitive Walkthroughs – Third Step • As the walkthrough is being done, a record of critical information is compiled in which: • The assumptions about what would cause problems and why are recorded. • Notes about side issues and design changes are made. • A summary of the results is compiled.

42. Steps involved in Cognitive Walkthroughs – Fourth Step • The design is then revised to fix the problems presented.

43. Pluralistic Walkthroughs

44. What is Pluralistic Walkthrough? • “Another type of walkthrough in which users, developers and usability experts work together to step through a scenario, discussing usability issues associated with dialog elements involved in the scenario steps” (Nielsen and Mack, 1994)

45. Steps of Doing Pluralistic Walkthroughs • Scenarios are developed in the form of a series of hard-copy screens • The scenarios are presented to the panel of evaluators. • The panelists discuss the actions that they suggested for that round of the review. • Then the panel moves on to the next round of screens.

46. What is benefit of Pluralistic Walkthrough? • Pluralistic walkthroughs have the advantage of providing a diverse range of skills and perspectives. • The result of pluralistic walkthroughs includes a strong focus on users’ tasks. • The approach also lends itself well to participatory design practices by involving a multidisciplinary team in which users play a key role.

47. Thank you~Any question?