CHAPTER 10 Introducing Evaluation

1. CHAPTER 10 Introducing Evaluation Doosup Baek Sungmin Cho Syed Omer Jan

2. Chapter Goals • Explain the concepts and terms used to discuss evaluation • Discuss HutchWorld Case Study • Examine different techniques used at different stages of development • Show and discuss how developers deal with real world problems and constraints

3. Definition of Evaluation ?? • It is the process of systematically collecting date that inform us about what it is like for a particular user or group of users to use a product for a particular task in a certain type of environment

4. Two main types of Evaluation • Formative Evaluation is done at different stages of development to check that the product meets users needs • Summative Evaluation assesses the quality of a finished product • Our focus is going to be on formative evaluation

5. What to Evaluate It’s a continuous process which examines: • Focus on users and their tasks • Observe, measure, and analyze their performance with the system • Design iteratively Important for designer to check and make sure that they understand user requirements

6. Why you need to Evaluate • Designers should not presume that everyone is like them or that the following of set guidelines would guarantee them good usability • Evaluation is needed to check that users can use the product and like it

7. Why you need to Evaluate (cont’d) • Bruce Tognazzini – Usability Consultant “Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain”

8. Why you need to Evaluate (cont’d) Toganazzini’s 5 resons to evaluate: • Problems are fixed before the product is shipped, not after • The team can concentrate on real problems, not imaginary ones • Engineers code is sharply reduced • Time to market is sharply reduced • Upon 1st release, your sales department has a rock solid design it can sell without having to pepper their pitches with how well the next release will work

9. Why you need to Evaluate (cont’d) • Usability testing involves measuring the performance of typical users on typical tasks • Satisfaction, can be evaluated through questionnaires and interviews • Trends are towards evaluating more subjective user-experience goals, like emotionally satisfying, motivating fun etc.

10. When to Evaluate • New product • Use mockups, sketches, and other low fidelity prototyping techniques are used to represent design ideas • Upgrading existing products • Compare user performance and attitudes and contrast new products with the previous versions • Evaluation is a key ingredient for a successful design

11. 1984 OMS • Background • Voice mail system for Olympic Games Contestants and their families could send and receive messages • Developed by IBM • Reason for intense evaluation • IBM’s reputation at stake • Olympics a high profile event

12. 1984 OMS (cont’d) • Evaluation activates during development • Use of printed scenarios for feedback • Iterative testing of user guides • Early simulations • Early demonstrations • Overseas test • “Try-and-Destroy-it” with CS students • Heavy traffic tests • Pre-Olympic field tests

13. HutchWorld Case Study • Virtual community • Collaboration • Microsoft’s Virtual Worlds Research Group • Fred Hutchinson Cancer Research Center • Uses • Chatting, storytelling, discussions and emotional support • Why ?? • Cancer patient isolation issues

14. HutchWorld Design Question ?? • Needs • Useful • Engaging • Easy to use • Emotional satisfaction • Early Ideas • What resource are available to patients? • What are the specific needs of the user’s? • What should it look like? • How will users interact within the Virtual community

15. Design Team Efforts !!! • Interviews with patients, caregivers, family, friends, clinicians, social support groups, former friends, and experts • Reading of latest research literature and HutchWorld web pages • Visiting Fred Hutch research facilities, and the Hutch school for pediatric patients and juvenile patient family members

16. Problems • Inadequate non-verbal feedback • Potential for mis-understanding • No : • Facial expressions • Body language • Tone of voice

17. Features of HutchWorld • Availability • Anytime, day or night • Regardless of geographic location • Design to resemble the outpatient facility • This real-world metaphor helped users infer the functionality • Synchronous chat environment was selected for realism • 3D photographic avatars

18. Testing HutchWorld • Test 1: • six computers • scaled-back prototype • Microsoft specialists trained Hutch volunteers • events were hosted in the prototype • Test 1 observations: • general usage of the prototype • usage of the space during unscheduled times

19. Testing HutchWorld (cont’d) • Test 1 results: • small user community • critical mass concept – not enough participants to fill the chat room for successful conversation • lack of interest • patient availability • patients preferred asynchronous communication (via email, journals, etc.) • prototype did not include original computer uses • patients played games and searched the internet

20. Redesigning HutchWorld • a more “unified” product was desired that included a variety of communication, information, and entertainment tasks • new support: • more asynchronous communication • information-retrieval tools • email, a bulletin board, text-chat • games and other entertainment tasks • a web page creation tool • a way to check if anyone is around to chat with

21. Usability Tests • Seven participants • four had used chat rooms • all had browsed the web • given five minutes to get familiar with software • A running commentary was given by each during exploration (what each was looking at, thinking, or confused by) • After five minutes, a series of structured tasks were given focusing on how the participants: • dealt with their virtual identity • communicated with others • retrieved desired information • found entertainment

22. Questionnaire • After the test participants were asked to fill out a questionnaire about their experience with HutchWorld • What did you like about HutchWorld? • What did you not like about HutchWorld? • What did you find confusing or difficult to use in HutchWorld? • How would you suggest improving HutchWorld?

23. Usability Findings • The back button did not always work. • Users ignored navigation buttons • more prominent buttons were needed • Users expected that objects in 3D would do something when clicked on • provide links to web pages when objects are clicked • Users did not realize other real people were interacting with them in the world • wording was changed in the overview description • Users did not notice the chat window and instead chatted with people on the participation list • instructions on where to chat were clarified

24. Future of HutchWorld • Evaluation of the effects of the software at the Fred Hutchinson Center • Investigation will include: • How the computers and software impact the social wellbeing of the patients and their caregivers? • What type of computer-based communication best supports this patient community? • What are the general usage patterns of the system? • How might any medical facility use computers and software like HutchWorld to provide social support for its patients and caregivers?

25. Discussion

26. Reference

27. Reference • The “Star Fire” Video Prototype Project • CHESS (Comprehensive Health Enhancement Support System) Module Demos • Asthma • Microsoft Social Computing Group • Olympic Message Service • The Schaffer Method of user-centered design • Chart • HutchWorld video clip