Think-aloud usability experiments or concurrent verbal accounts

Judy Kay CHAI: Computer human adapted interaction research group School of Information Technologies Think-aloud usability experiments or concurrent verbal accounts

Overview Empirical methods Think-aloud Tasks Abstract versus concrete Task design and avoiding leading the user Benefits Disadvantages Naturalised think-aloud

Postconditions for this week Describe the uses of Think Alouds Describe the processes for conducting one Describe advantages and limitations Ability to use Think-aloud as relevant and for your project Defining concrete tasks, that assess interface well Recruiting participants (who, how many) Conducting the think-aloud Justify the use of Think Aloud in the overall testing of a pervasive computing application

Think aloud protocols Ask user to “think aloud” as they use the interface Often used with video, audio taping Otherwise MUST make notes Helps observer interpret what is going on Gives qualitative data mainly 3-5 users may be enough (Nielsen)‏ for each stage of refining the prototype

Case study Test usability of teacher’s data projection facilities in this classroom What is intended functionality? Formulate some concrete tasks – write these as instructions eg You want to display the screen of your mobile phone to the whole class.

Case study Work in your groups to identify 3 tasks relevant to testing the teacher’s use of projection facilities in this room

Case study • Call for volunteers to be users for this trial • Have not used these facilities before • Have used them in other places in the uni • Call for volunteers to conduct the trial • All will make helpful notes

Classification of think-aloud formative versus summative predictive v empirical laboratory v naturalistic qualitative v quantitative For the project, you goal will be?

Design cycle User Centred Design Define criteria for success Define concrete tasks users should be able to do - use these in evaluation Prototype construction Usability study Decide just what data to collect Test design of experiment for timing (trial it)‏ Recruit users Run study Goto top

Recruiting users How representative are they? similarity to intended user population Age Gender experience in area interest/motivation computer literacy What effect does user population have for conclusions?

Pitfalls Defining the right concrete tasks Test all key aspects Multiple tasks for same aspects Instructions to the users Do NOT lead the user Take particular care not to use words that are identical to terms on the interface

Single Sentence Statement for lecture room control system • Lecturers, without prior training, can use the control interface to make effective use of the lecture room facilities, as needed in a lecture (within 2 minutes).

Requirements • Lecturers, can, without prior training, and within 2 minutes, use the control interface to: • Display a physical resource (such as a sheet of paper) to the class • Increase the audio level of the lectern microphone • Set the lighting for showing a video • Raise the particular screen wanted • …

Example tasks (sparse word for OHP) • Assume: • initial state of theatre is with data projectors showing internal computer. • props: a mobile phone the user is asked to pretend they brought to the class, book • lecture theatre has a whiteboard behind the screens • Show the class the screen of your mobile phone. • You want to write on the left whiteboard (student perspective). • You want to alter the lighting to show a video to best effect. • You want to now show this book on both screens.

Example task design Requirement

Benefits of think aloud “show what users are doing and why they are doing it while they are doing it in order to avoid later rationalisations” (Nielsen, Usability Engineering, Academic press 1993, p195)‏ Cheap Slows users down studies show users may work faster with fewer errors due to care on critical elements

Problems of think aloud Not directly quantitative Add cognitive load to users User's “theories” must be interpreted with care Slows users down Users are aware they are being observed so behave accordingly

Facilitating think aloud What are you thinking now? What do you think that message means? (only after the user has noticed the message and is clearly spending time on it)‏ don't help user except with How do you think you can do it? if user appears surprised, Is that what you expected to happen?

Problems with observing users Hawthorn effect People rationalise (Telling more than we can know)‏ Qualitative data

Next lab activity – lodge tasks The group will work together to formulate three concrete tasks for testing prototypes You will store these And share them with the class

Summary Relatively inexpensive Can identify major flaws And may indicate causes of user problems May give access to user's mental model Alters activity => meaningfulness

Think-aloud usability experiments or concurrent verbal accounts