HCI 2009-2010 Week 3 - Evaluation

1. HCI 2009-2010Week 3 - Evaluation

2. Breaking news… • Brain computer interaction (Radio 4, 6/10/09, 0720) • Research relating to mind input. • Seen as of possible use for people with disabilities

3. More breaking news… • … seeking student reps with the aim of achieving one rep in each personal tutor group. We did not get a response from the level 6 IT Systems for Business students so their views will be unheard unless someone comes forward before this week’s deadline

4. Introduction – Why Evaluate? • Dix et al (2004) argue that evaluation has these main goals: • Assess the extent and accessibility of the system’s functionality • Assess users’ experience of the interaction GOW • Identify any specific problems with the system. • Save money

5. Some claims in respect of usability evaluations (courtesy of T Renshaw) • AOL • Improvements to web architecture following usability testing increased traffic to second level web areas by 120% • IBM • Improvement in web usability led to 400% increase in sales • Usability evaluation reduced time to complete task by 9.6 minutes leading to internal savings of $6.8million • Ford • Usability improvements to accounting system for Ford dealerships removed need for user support with savings of $100,000 • Sun Microsystems • $20K invested in usability can yield returns of $152 million - $7500 per dollar of investment

6. Two main approaches to evaluation • To try to achieve these goals there are, it can be argued, 2 broad approaches to evaluation of interactive systems: • Predictive evaluation “Expert” evaluation • Usability testing Evaluation involving users

7. PREDICTIVE EVALUATION Typically we’re here dealing with a design or prototype system. Possible techniques: a) Task Action Language Evaluates a command language for consistency and simplicity Examples of command language: DOS UNIX Considers number of rules in the grammar needed to achieve a particular task

8. PREDICTIVE EVALUATION b) GOMS (Goals, Operators, Methods, Selection rules) Analyses a task Provides time-based predictions of how a user will carry them out Cognitive Complexity Theory builds on GOMS predicts complexity and learnability of the specified user interface

9. PREDICTIVE EVALUATION c) Expert walk-through User interface expert goes through the system Identifies what s/he sees as potential problems Can use independent experts GOW Possible strengths: quick and easy Possible drawbacks: May be hard to find an expert Technique relies on expert’s judgement

10. PREDICTIVE EVALUATION d) Cognitive walk-through Similar to expert walk-through, but involves evaluators going through steps of various tasks to consider usability • See Dix et al (2004) pp 321/2 (at back of handout)

11. PREDICTIVE EVALUATION e) Heuristic evaluation Several evaluators independently critique a system. The system is tested against 10 heuristics – see back of lecture handout (Dix et al pp 324-6) http://www.useit.com/papers/heuristic/heuristic_list.html

12. USABILITY TESTING • Here we are evaluating with actual users. • “In theory there is no difference between theory and practice. In practice there is.” Yogi Berra • We could use a design or paper prototype or a running program. • Approaches:

13. USABILITY TESTING a) “Simulation” • Printed or on-screen version of displays used in pilot tests (“usability lab testing” – see figure 1 below)

14. USABILITY TESTING • Wizard of Oz In effect, a realistic but simulated (“pretend”) version of the system is evaluated.

15. USABILITY TESTING b) Questionnaire • Use them to: • identify features users find hard, and/or • assess users’ attitude to the design • Johnson (1992): • questionnaires are poor at identifying usability or reasons why design is unusable • however • specific user interface have been developed • and Shneiderman and Plaisant (2005) favour them

16. USABILITY TESTING c) Observation • video tape users performing tasks at the interface, and review for errors etc. • can have two users working together • can invite users to “think aloud” (a “user walkthrough” (Le Peuple and Scane 2003)) • similar technique – “co-operative evaluation” [gow] • Possible problem – “observer effect” • Aka “Hawthorne effect” • The knowledge that people are being observed may make them act differently • And clandestine observation is not allowed

17. USABILITY TESTING d) Interviews and group discussions • Can use former to pursue specific issues of concern • The latter can establish the universality of comments • “Focus groups” e) on-line consultations • e.g. • suggestion box • trouble reporting • bulletin board • user news letter

18. USABILITY TESTING f) Continuous user performance data collection • (Software keeps a log of user actions (Shneiderman 2002)) • e.g. data concerning • patterns of system usage • rate of errors • specific errors

19. USABILITY TESTING g) Controlled experimentation • Use an experiment to evaluate an implemented design • Steps in designing an HCI experiment: 1) Formulate hypothesis adding video to my CBL will aid learning 2) Develop predictions from the hypothesis the average user will learn more facts with video than without 3) Choose a means to test the predictions eg a test at the end of the experiment 4) Identify all variables that might affect the results of the experiment ability of students, nature of domain, type of machine......

20. 5) Decide which are • independent variables • use of video • dependent variables • performance of the students • variables that need to be controlled out • time of day, type of machine, .....

21. 6) Design the experimental task and method • 7) Select participants • 8) Decide the • experimental design • eg between groups, within groups, longitudinal • data collection method • eg observation, written test,.... • means of controlling out confounding variables • 9) Decide on statistical or other analysis • 10) Carry out pilot study

22. As is often the case, there are arguments for and against experiments in HCI. • See extract in tutorial notes from Hobbs and Moore (1998) • One approach to experimentation = eye tracking (Web and Renshaw 2008) – next week - vital for the assignment

23. CHOICE OF EVALUATION TECHNIQUE(S) • Clearly, then, there are a number of evaluation techniques that can be used. • How to select the appropriate diet of techniques is an interesting and important question. • “The choice of evaluation methodology … must arise and be appropriate for the actual problem, research question or product under consideration” (Greenberg and Buxton 2008)

24. CHOICE OF EVALUATION TECHNIQUE(S) • See extract in tutorial notes from Hobbs and Moore (1998) for some thoughts on this • A common view – use both predictive evaluation and usability testing

25. Requirements Specification • When to evaluate? implementation The STAR Model Evaluation Task analysis Prototyping Conceptual Design

26. SUMMARY • A variety of evaluation techniques exist, and are being researched and developed • Should use one or more to inform future design • Johnson (1992) - treat evaluation results as over-generous rather than as over-critical • Please get this lecture out when working on the HCI assignment! • And PIPE, probably!