Evaluating User Experience Methods: Heuristic vs. Cognitive Walkthroughs vs. Usability Testing

1. A Comparison ofHeuristic Evaluation, Cognitive Walkthroughs and Usability Testing Alfred Kobsa University of California, Irvine

2. Heuristic Evaluation • Normally performed by a team of 3-5 experts • Original designers are do not see their own mistakes easily • Experts go through guidelines (heuristics) • Rate individually, form consensus • Can be performed on early designs, mockups and prototypes • Pros: • Cheaper than usability experiments • Reasonably effective: • Rule of thumb: 1 expert ca. 40% of errors, 2: 50%, 3: 60%, 5: 80% • Cons: • HCI experts find mostly local problems (when domain expertise is also present, more and more global problems will be found)

3. Cognitive Walkthrough • Performed by original designers or experts • (individuals or groups) • Analyst(s) imagine performing a task and walk through design documents • Analysts try to determine whether users will be able figure out where to go and how to do things in the design • Can be performed based on early designs, mockups and prototypes • Pros: • Cheaper than usability experiments • Reveals global errors more readily than heuristic evaluation • Cons: • Original designers do not see their own mistakes easily • Experts may need long to understand the task at hand

4. Usability testing • Performed by original designers (or experts) • with users who are representative for the target population • planned in teams, carried out by (parallel) subteams of 1-4 people • Pros: • Reveals more usability problems than other methods • Finds more global problems • Finds more unique problems (?) • Cons: • Time-intensive: several weeks to a few months • Does not detect local/minor problems very well • ☛ perform heuristic evaluation and/or cognitive walkthrough first