Video Tape Evaluation and Demonstarion

1. Video Tape Evaluation and Demonstarion Mohammod Shamim Hossain

2. Details: Videotaped Evaluation A software engineer studies users who are actively using the user interface To observe what problems they have Rather than to measure numbers The sessions are videotaped Can be done in user’s environment Activities of the user: Performs pre-defined tasks With or without detailed instructions on how to perform them Preferably talks to herself/himself as if alone in a room Yields ‘think-aloud protocol’ This process is called ‘co-operative’ evaluation when the software engineering and user talk to each other

3. The importance of video: Without it, ‘you see what you want to see’ You interpret what you see based on your mental model In the ‘heat of the moment’ you miss many things Minor details (e.g. body language) captured You can repeatedly analyze, looking for different problems Tips for using video: Several cameras are useful Software is available to help analyse video by dividing into segments and labeling the segments Evaluation can be time consuming so plan it carefully Details: Videotaped Evaluation

4. Select 6 to 8 representative users per user class E.g. client, salesperson, manager, accounts receivable Invite them to individual sessions Sessions should last 30-90 minutes Schedule 4-6 per day If system involves user's clients in the interaction: Have users bring important clients or have staff pretend to be clients Select facilitators/observers and notetakers Prepare tasks: Select the most commonly used tasks plus a few less important tasks Write task instructions for users Estimate the time it will take to complete each task plus extra time for discussion Prepare notebook or form for organizing notes Steps for videotaped evaluation

5. Set up and test equipment Hardware on which to run system Audio or video recorder Software logs Do a dry run (pilot study)! At the Start of an Observation Session explain: nature of project anticipated user contributions why user's views are important focus is on evaluating the user interface, not evaluating the user all notes, logs, etc., are confidential user can withdraw at any time usage of devices relax! Sign informed consent form: very important Steps for videotaped evaluation

6. Start user verbalizing as they perform each task (thinking aloud) For co-operative evaluation, software engineer also verbalizes Appropriate questions to be posed by the observing software engineer: Steps for videotaped evaluation

7. Hold a wrap-up interview (de-briefing) What were the most significant problems? What was most difficult to learn? Etc. Analyze the videotape to find malfunctions Lab exercise: Videotaped evaluation of a software product Steps for videotaped evaluation

8. Assignment : Videotaped Cooperative Interface Evaluation You are expecting to do the videotaped user interface evaluation for some functionality of Corel Photo House 3 software. The co-operative UI evaluation should be based on the following tasks as for Example: Task 1 (simple & Easy) Instruction for Evaluation of “Corel Photo House 3”  First Image (demon) Create a blank image Add “Cutout” of Dog on that image (Name of the Cutout: Puppy) Draw red horns on the dog Change the image so that it becomes a range of red Task 2 (Medium) Task 3 (little bit Complex)

9. Report A summary of the procedures you used to do the evaluation (5 Marks) When, where and how did you do the evaluation process? What did the subject do to achieve the task, Pseudo code like description of the steps that have been taken by each user to accomplish the required task. What happened as the evaluation proceeded? Here you should provide sufficient detail so the marker can see that you followed good procedures and handled procedural problems well. A complete list of malfunctions that you found (1 per line) (5 marks). A discussion of the four most significant malfunctions (5 marks). For each provide the following, An excerpt of the protocol. i.e. a verbatim transcript of 5-15 lines describing what the user did and said, what you said and what happened (around the time the malfunction occurred) You can embellish this with a picture illustrating the malfunction if this makes it clearer The result of malfunction analysis. Brief recommendations for the changes. Conclusion

10. General Notes Do not forget to sign the Informed Consent Form, available on the course web site. For malfunction analysis follow (in detail) the procedures outlined in following slides Remember to do a short dry run (pilot study) so you become comfortable with the procedures and A-V equipment. The dry run must use a different task from the main session. Remember that co-operative evaluation requires both you and the subject user to verbalize. Your subject(s) should not be someone intimately familiar with the software (i.e. not a designer); however the subject should know or be taught the basics of the system. Total videotaping time should be 20-30 minutes Videotape the session (the TA will help with this if needed) You can arrange to borrow cameras from A-V services, although if you do your study with the TA, she can take care of this for several groups at once. Do not hand in the tape with your report, but keep it in case the professor wants to see it. Erase the tape once you get your mark.

11. A disciplined approach to analyzing malfunctions Provides feedback into the redesign process Play protocol, searching for malfunctions Answer four distinct questions: Q1. How is the malfunction manifested? What do you notice and who noticed it? Q2. At what stage in the interaction is it occurring? Goal forming, action decision, action execution, interpretation of results Q3. At what level of the user interface is it occurring? Physical element level to task level Q4. Why is it occurring? What is its root cause List and prioritize possible cures Malfunction Analysis

12. Q1. How is the malfunction manifested? a) Malfunctions detected by the system (easiest to detect) omission of an argument incorrect date format Cure: Better prompts, consistency, visible examples, more forgiving of alternatives b) Malfunctions detected by the user during operation taking wrong path in menu hierarchy not finding required help not being able to perform a certain action not being able to tell which state system is in Cure: Improve functionality, feedback, clarity, simplicity

13. Q1. How is the malfunction manifested? c) Malfunctions undetected (until later) output produced is wrong due to wrong inputs unnecessary work performed Cure: Improve feedback indicating consequences of input; simplify d) Inefficiencies excessive response time excessive think time unnecessarily long command sequences unnecessary repetitions complex operations that require use of reference Cure: Simplify, speed system up

14. a) When the user decides on next goal (Forms an intent to do inappropriate thing) decides to empty a field because user thinks it is unimportant (when it is important) decides to charge default exchange rate (when should obtain current exchange rate) Cure: Lead user through task better; better feedback; better training b) When the user specifies the action (Action does not match the goal) deletes the record instead of emptying a field charge reciprocal of exchange rate Cure: Improve clarity, feedback, prompts, conceptual model

15. c) When the system executes the action Defects in functionality Cure: Fix functionality in normal way d) When the user interprets the resulting system state thinks bank account has been debited when it has not thinks system has ‘hung’ when it has not thinks some data must be entered when it is the default cannot understand resulting error message Cure: Better feedback, better conceptual model

16. a) Task level (Task and goals not supported) What the user wants to do cannot be done by the system Functionality is not provided Cure: Add functionality b) Conceptual level (User has wrong mental model; does not understand intended conceptual model) thinks that money is being deducted from bank account when it is being charged to a credit card thinks that dragging a file to the desktop means they are no longer on the disk thinks that dragging a disk to the trash can icon deletes disk contents Cure: make conceptual model clearer; improve metaphors

17. c) Interaction style level (system wide problem) does not know how to pull down a menu scrolls a page instead of a line goes to next screen instead of scrolling retypes command after an error instead of editing it Cure: make operation of the interface more intuitive and consistent d) Interaction element level (specific detail inappropriate) selects wrong button because label is misinterpreted specifies invalid command syntax specifies wrong code for option Cure: More attention to details of the interface, simplification

18. e) Physical element level (Physical execution incorrect) presses wrong key accidentally clicks on wrong pixel in image out-types machine (actions lost) types ahead when system is computing; keystrokes later applied to wrong action Cure: Defenses to protect user from consequences; better hardware design; fix bugs in code

19. Lack of (on the part of the user): Motivation: Poor job satisfaction Attention: User is pre-occupied with other things. Input information processing: No feedback provided to tell user what is going on or cues provided by the system are not recognized or cues are misinterpreted Cures: Clearer, more consistent feedback Discrimination: user is unable to tell certain things apart e.g. red/green colour discrimination e.g. two icons that are similar Cures: Improved expression of information

20. Physical coordination: e.g. wrong item selected because of difficulty positioning cursor with mouse. Cures: Alternate interaction mechanisms, better feedback Recall: User did not remember command , syntax etc. Cures: Better mnemonics, online help, quick lookup  mechanisms, command completion Knowledge / lack of learning: User does not have business or software knowledge to make right choice.

21. Learning difficulties that cause malfunctions: Learning is difficult users get frustrated learning takes time; can be hard to apply Learners make ad-hoc interpretations they may not recognize their problem they may falsely think they have a problem Learners generalize from what they know they assume computers work like manual methods they assume consistency Learners have trouble following directions they often ignore them even if they see them they do not easily understand them

22. Problems and features interact they do not see that one problem can cause another Prerequisites and side-effects confuse learners Help facilities do not always help they do not know what to ask for too much detail is often provided Other causes of malfunctions: Excessive resource demands External events (e.g. noise) Misleading or inadequate training Unrealistic task definitions Intrinsic human variability