UBICOMP 2002 W2: User Centred Evaluations for Ubiquitous Computing Systems: Best Known Methods

1. UBICOMP 2002W2: User Centred Evaluations for Ubiquitous Computing Systems: Best Known Methods Scenario Evaluations for Ubiquitous Computing - Stories Come True VTT Information Technology, Human-Centred Design, Tampere, Finland

2. One ordinary morning in the near future You live in an apartment at the old town of Gothenburg, Sweden. It is an ordinary autumnal Tuesday morning. Your electronic calendar has noticed that you have to be at the office by 9, because you have a meeting at 9.15. Your car's travelling program has sent a message to calendar, that the trip from your apartment to the office takes 20 minutes. The calendar has sent a message to the alarm system, which actives wake-up at 8 o'clock. First there are silent voices, birds are singing softly in the background. Your favourite bird is finch. Little by little the curtains are opened, lights turn brighter and the finch's song becomes louder. As the awakening continues, the room fills with coffee smell and with the scent of coffee and with sounds of morning routines. Gradually you start to be aware of your surroundings and sit up at the side of the bed. You notice that you have a morning meeting and you go to the shower...

3. SCENARIO METHOD - definitions - what? • Scenarios = stories or descriptions of possible futures. • Story of the usage of a service or product. • Users evaluate scenarios made by others, or • Users build scenarios themselves.

4. SCENARIO METHOD - definitions - why? ( also benefits) • Scenarios help potential users to understand imagined applications and services of new technologies. • Scenarios treat technology quite flexibly: technology can either be described in detail or the focus can be more on the context of use • Scenarios make ideas more concrete for the purpose of analysis and communication.

5. SCENARIO METHOD - definitions - when? Human-Centred Design Process Literature study Analysis/evaluation of corresponding products Observation, interviews Initial context of use Initial user requirements New and revised requirements Scenarios Paper protos Mockups Design requirements Evaluation Revised design Design and implementation Prototypes Easy to use and useful product

6. The goals of the KEN project The main goal is to make sure that NAVI projects take into account user needs and preferences as well as ethical issues in the design. Results: 1) usability guidelines and tools 2) usability criteria 3) trade descriptions for personal navigation products 4) ethical guidelines 5) a study on usage cultures in Finland and in other countries 6) guidelines for internationalisation and localisation 7) international network with usability research organisations

7. Purpose of scenario evaluations • study how credible and useful potential customers considered the scenarios made by experts. SWOT analysis. • identify needs of the user groups for navigation services and let the participators innovate new navigation services • help people to understand the idea of personal navigation and then to elicit the ideas, attitudes, opinions and needs of different user groups with regard to navigation services and products. • timing: spring 2001

8. PIIP! Tarzan is already at the gates of the zoo when his navigator rings up. Tarzan has allowed himself to be located and his friend Clayton has noticed in his device that Tarzan has arrived to town. So Clayton rings his friend and suggests meeting with Tarzan as he is nearby..... Tarzan schedules to meet with Clayton in the afternoon. At the same time Tarzan allows advertisers to send word to him about nearby flower shops and their offerings because he wants to buy flowers for Jane.

9. Future Home -scenario evaluation • Future Home is a project which was set up to define a home network concept that will be based on short range wireless technologies. The beginning of the scenario evaluation • At first every project group member gave their own idea about what Future Home is. • We received 14 scenarios. We analysed them and found several overlaps. We stripped all scenarios into a flow of events and produced 5 scenarios: Services, Nomadic users, Monitoring and controlling the system, Entertainment and spare time, Communication. • Now the problem was, that it was not realistic to implement some of the scenarios during this project. Also there were far too many features presented. • The conclusion was to set up a vote in order to reduce the features: One partner / one vote. There were problems with this.

10. Future Home -Scenario evaluation continues • Our goal was still to find possible features and to build up final scenarios in order to clarify what the system prototype will do. • Other goal was to get the potential users' viewpoint in order to produce user requirements later on. How users accept the general concept of Future Home and Ubicomp. • Also real users feedback may affect the technical requirements. Interviews: users, provides, usability • 20 people were interviewed during February 2002, 10 in Finland and other ten in Germany. • The 5 scenarios were sent to them beforehand and each interview took approximately an hour. The users were interviewed at their homes usually in the evening. • Concentrated on following issues: ...

11. Scenario material • Demonstration video was presented from the Internet: http://cooltown.hp.com/cooltownhome/cooltown-video.asp • We had visualiations about the possible terminal devices made by Nokia, they were presented as the scenario was discussed with the user. • The scenario stories were posted or given to the users a week before the actual interview. About the interviews • All interviews were made at the user’s home or at the office, either in the evening after work or at week-end.

12. Future Home -Scenario evaluation results • most wanted vs. rejected features were listed. • The results were reported to the project group and they were also added to a Deliverable -paper. • A requirement specification was written and the scenarios were presented in a new formal table form.

13. Evaluating scenario evaluations - problems scenario building: - terminology -coherent / uniform structure of scenarios, e.g. flow of events, length of scenarios, actors - potential user participation qualitative nature of the evaluations: - many interpretations are possible from the same material - how to ensure that different evaluators are doing the same (that results may be compared) structured interviews, planning and rehearsing together? - leading group discussions requires training and special skills - planning, doing and analysing could be time consuming - results are qualitative: answering to the question why and not how many - quantitative methods are also needed presenting the results: - to peer groups - how to ensure that the results of the evaluations (user requirements and attitudes) are put to use later on

14. Evaluating scenario evaluations -development ideas 1. Could it be possible to collect some valid quantitative material, e.g. with web forms? 2. More active user participation in scenario building: - more ethnography: diaries, participant observation - more mock-up prototyping with the users in authentic settings - how to ensure user commitment and motivation: money? 3. Scenario evaluations: more interaction with users - user’s own habits and experiences, use of senses, interactive scenarios......