Cognitive Walkthroughs and heuristic evaluation

1. Cognitive Walkthroughs and heuristic evaluation

2. Evaluation - definitions • to assess the extent of the system’s functionality (i.e. does it map onto the user’s task requirements) • to assess the effect (not affect, generally) of UI on the user • to identify any specific problems with the system

3. Evaluation is ... • Essential • Time consuming • Often difficult to organise • e.g. the availability of subjects / participants • But there is an easy(ish) alternative to working with end-users

4. Cognitive walkthroughs • Structured approach to usability analysis • Intended to supplement NOT replace empirical approaches (empirical = with users in this instance) • A number of different versions exist... • ‘Hand simulation of the cognitive activities of a user’ Polson et al, 1992

5. Performing a walkthrough • Create a usage scenario • Identify a task within the scenario • Perform the walkthrough • Record the problems found

6. To begin... • Scenario: • e.g. someone writing a letter • Task: • To copy and paste some text • sub-tasks • Copy text into clipboard • Select point to enter (pasted) text • Issue paste command

7. Tasks for copy & pasting text Note the order!

8. Examining the sub-goals • Will the user try to achieve the right goal? • What knowledge is needed to achieve the right subgoal? Will the user have this knowledge?

9. Satisfying goals • Will the user notice that the correct action is available? • Will the user associate the correct action with the sub-goal they are trying to achieve?

10. Feedback • Will the user perceive the feedback? • Will the user understand the feedback? • Will the user see that progress is being made towards the solution of their task?

11. Tasks for copy & pasting text 1. Apply Q1& 2 to sub-goal B 2. Apply Q1 & 2 to C 3. Apply Q3-7

12. Recording problems • Any questions answered to the negative indicate potential usability problems • Describe the problem in as much detail as possible • When the problem occurred • Why the problem occurred • The consequences of the problem

13. Example - setting column widths

19. Now your turn... • Using your task analysis of requesting a loan conduct a cognitive walkthrough • Report the problems • consider how they might be fixed

20. heuristic evaluation ‘heuristic’ - ‘used of problem solving techniques that proceed by trial and error’ (related to Greek ‘eureka’) Longman Concise English Dictionary, 1985 a method of usability evaluation where an analyst finds usability problems by checking the user interface against a set of supplied heuristics or principles Lavery, Cockton and Atkinson, 1996

21. who should do heuristic evaluation? • use more than one evaluator • ideally should not be the designer • ideally should be a usability specialist • technical authors also useful • each carries out independent inspection, then aggregate findings • evaluators may need help • unless ‘walk-up-and-use’ application • could provide typical usage scenario

22. effectiveness of increasing number of evaluators (after Nielsen, 1993)

23. evaluator performance Nielsen 1992: • same interface evaluated by 3 groups • novice (knowledge about computers only) • usability experts • usability and specialist domain experts

24. how to do heuristic evaluation • go through interface, compare against recognised usability principles • first pass flow of interaction and general scope • subsequently focus on specific elements • typically 1 - 2 hours in total • output: list of usability problems cross-referenced to usability guidelines

25. learnability memorability errors efficiency subjective satisfaction usability metrics

26. but how to set the target level... • skill and intuition... • better than last version • better than the competition • client targets • set a range of levels • unacceptable • minimum • target • ideal

27. a usability profile

28. STOP remember materials for next week

29. usability testing - planning • draw up a test plan • see separate handout ‘Checklist for usability test plans’ • informal use by the test team • formal use for QA procedures • consider whether to use video/audio recording

30. usability testing - users • test users should represent target users • remember sales staff as a special user group • may need to give basic training • getting hold of users • internal users should be easy • customers from user groups may help • paid volunteers : students, classified ads.. • take account of older users if relevant

31. pay-off ratio for user testing(after Nielsen, 1993)

32. usability testing - designing test tasks • representative & provide reasonable coverage • do-able but not trivial • consider relating to a larger scenario • provide a written task description • present in increasing level of difficulty • decide whether to use verbal protocols

33. relative effectiveness • Karat el (1992) • expert individual & group walkthroughs • used guidelines and tasks • usability testing • users identified and described problems • testing identified most problems • including some severe ones missed by experts

34. effectiveness continued • walkthroughs useful when resources linited, or for early design • team walkthroughs better than individuals • techniques are complementary • cost effectiveness similar • also formal experimental trials

35. usability testing - procedure • preparation • remember to switch off screen-savers, email, etc. • introduction • testing • debriefing • questionnaires if used • also ask about the testing process • write up quickly

36. learnability time to reach specified level of proficiency, e.g. complete a specified, representative task note that learning is a continuum memorability test users on commands after trial session errors number of errors in completing specified task subjective satisfaction rating scales physiological measures efficiency times for experts to complete specified task(s) frequency of ‘non-productive’ actions ratio of used to unused commands usability metrics