1 / 18

Human Computer Interaction

Human Computer Interaction. Introducing evaluation. The aims. Discuss how developers cope with real-world constraints Explain the concepts and terms used to discuss evaluation Examine how different techniques are used at different stages of development. Two main types of evaluation.

ruth-perez
Download Presentation

Human Computer Interaction

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Human Computer Interaction Introducing evaluation

  2. The aims • Discuss how developers cope with real-world constraints • Explain the concepts and terms used to discuss evaluation • Examine how different techniques are used at different stages of development

  3. Two main types of evaluation • Formative evaluation is done at different stages of development to check that the product meets users’ needs • Summative evaluation assesses the quality of a finished product • Our focus is on formative evaluation

  4. What to evaluate • Iterative design & evaluation is a continuous process that examines: • Early ideas for conceptual model • Early prototypes of the new system • Later, more complete prototypes • Designers need to check that they understand users’ requirements

  5. Bruce Tognazzini tells you why you need to evaluate • “Iterative design, with its repeating cycle of design and testing, is the only validated methodology in existence that will consistently produce successful results. If you don’t have user-testing as an integral part of your design process you are going to throw buckets of money down the drain.” • See AskTog.com for topical discussion about design and evaluation

  6. When to evaluate • Throughout design • From the first descriptions, sketches etc. of users needs through to the final product • Design proceeds through iterative cycles of ‘design-test-redesign’ • Evaluation is a key ingredient for a successful design

  7. Approaches: Naturalistic • Naturalistic: • describes an ongoing process as it evolves over time • observation occurs in realistic setting • ecologically valid • “real life” • External validity • degree to which research results applies to real situations

  8. Approaches: Experimental • Experimental • study relations by manipulating one or more independent variables • experimenter controls all environmental factors • observe effect on one or more dependent variables • Internal validity • confidence that we have in our explanation of experimental results • Trade-off: Natural vs Experimental • precision and direct control over experimental design versus • desire for maximum generalizability in real life situations

  9. Approaches: Reliability Concerns • Would the same results be achieved if the test were repeated? • Problem: individual differences: • best user 10x faster than slowest • best 25% of users ~2x faster than slowest 25% • Partial Solution • reasonable number and range of users tested • statistics provide confidence intervals of test results • 95% confident that mean time to perform task X is 4.5+/-0.2 minutes means95% chance true mean is between 4.3 and 4.7, 5% chance its outside that

  10. Approaches: Validity Concerns • Does the test measure something of relevance to usability of real products in real use outside of lab? • Some typical reliability problems of testing vs real use • non-typical users tested • tasks are not typical tasks • physical environment different • quiet lab vs very noisy open offices vs interruptions • social influences different • motivation towards experimenter vs motivation towards boss • Partial Solution • use real users • tasks from task-centered system design • environment similar to real situation

  11. Qualitative Evaluation Techniques

  12. Qualitative methods for usability evaluation • Qualitative: • produces a description, usually in non-numeric terms • may be subjective • Methods • Introspection • Extracting the conceptual model • Direct observation • simple observation • think-aloud • constructive interaction • Query via interviews and questionnaires • Continuous evaluation via user feedback and field studies

  13. Querying Users via Interviews • Excellent for pursuing specific issues • vary questions to suit the context • probe more deeply on interesting issues as they arise • good for exploratory studies via open-ended questioning • often leads to specific constructive suggestions • Problems: • accounts are subjective • time consuming • evaluator can easily bias the interview • prone to rationalization of events/thoughts by user • user’s reconstruction may be wrong

  14. Evaluating the 1984 OMS • Early tests of printed scenarios & user guides • Early simulations of telephone keypad • An Olympian joined team to provide feedback • Interviews & demos with Olympians outside US • Overseas interface tests with friends and family. • Free coffee and donut tests • Usability tests with 100 participants. • A ‘try to destroy it’ test • Pre-Olympic field-test at an international event • Reliability of the system with heavy traffic

  15. Development of HutchWorld • Many informal meetings with patients, carers & medical staff early in design • Early prototype was informally tested on site • Designers learned a lot e.g. • language of designers & users was different • asynchronous communication was also needed • Redesigned to produce the portal version

  16. Usability testing • User tasks investigated: • how users’ identify was represented • communication • information searching • entertainment • User satisfaction questionnaire • Triangulation to get different perspectives

  17. Findings from the usability test • The back button didn’t always work • Users didn’t pay attention to navigation buttons • Users expected all objects in the 3-D view to be clickable • Users did not realize that there could be others in the 3-D world with whom to chat • Users tried to chat to the participant list

  18. Key points • Evaluation & design are closely integrated in user-centered design • Some of the same techniques are used in evaluation & requirements but they are used differently (e.g., interviews & questionnaires) • Triangulation involves using a combination of techniques to gain different perspectives • Dealing with constraints is an important skill for evaluators to develop

More Related