1 / 25

Evaluation of usability tests

Evaluation of usability tests. Why evaluate?. choose the most suitable data-collection techniques identify methodological strength and weaknesses of a user test . Evaluation Criteria for data-collection techniques. Utility how useful are the data? Costs resources needed? Objectivity

ghada
Download Presentation

Evaluation of usability tests

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Evaluation ofusability tests

  2. Why evaluate? • choose the most suitable data-collection techniques • identify methodological strength and weaknesses of a user test

  3. Evaluation Criteria fordata-collection techniques • Utility • how useful are the data? • Costs • resources needed? • Objectivity • how much subjective judgement is involved? • Level of detail • is the amount and resolution of the data suitable? • Intrusiveness • does the method interfere with the user’s performance?

  4. Strengths: Level of detail: Allows you to experience the context in which performance takes place Weaknesses: Level of detail: Difficult to keep up with the pace of the user Objective: Based on your own subjective judgement as an observer Observations in real time

  5. Strengths: Utility: Allows you to conduct detailed analysis of various usability attributes Utility: Can obtain data about the user’s reasoning (”Think-aloud”) Weaknesses: Costs: Time consuming Utility: Lots of data not being used Intrusiveness: ”Think-aloud” may disturb the user Observations from video

  6. Observations: Real time or Video? Real time Video Context Product Context Product Level of detail

  7. Strengths: Objective: The data are collected automatically Costs: Automated data collection requires little effort from the test team Weaknesses: Level of detail: Both the amount of data and the resolution can be too high Utility: It can be difficult to create useful measures Event logs

  8. http://zing.ncsl.nist.gov/WebTools/VisVIP/overview.html

  9. Strengths: Level of detail: Can be tailored to fit the purpose of the test Utility: Can be used in several setting with different products Costs: It doesn’t take long time to develop Weaknesses: Objectivity: Based on subjective judgement Utility: Difficult to construct good items Questionnaire, self-made

  10. Strengths: Utility: Can be used in several setting with different products Costs: the data are typically easy to transform into measures Weaknesses: Level of detail: Validated questionnaires may not address the features of the interface you are interested in. Objectivity: based on subjective judgement Questionnaire, validated

  11. Summary data-collection techniques The assessment concern MEASURES and not use/problem descriptions; ++ = very good; + = good; - = not so good; -- = poor

  12. Observation and Interviews are the most suitable data-collection techniques for use/problem descriptions …Use/problem descriptions

  13. Evaluation of measures • The evaluation criteria of thedata-collection techniques • Validitity • Reliability

  14. Validity Do you measure what you believe you measure?

  15. Reliability Do you obtain the same results when you measure the same thing during similar conditions at different points in time?

  16. Relationship betweenValidity & Reliability • Evaluating the validity of a measure is primarily based on subjective judgement, while reliability is typically evaluated by means of statistics • It is possible to obtain reliable results that are invalid, but not unreliable results that are valid!

  17. How can you avoid invalid results? • Use several measures! • Triangulation • Multiple operationalism

  18. Ethical issues • Be well prepared - act professionally! • Create a script • Introduction • During test • Debriefing • Create a consent form

  19. Ethical issues • The product is being tested, not the user! • Respectful treatment: preserve integrity • Informed consent • Inform the user what will happen, how the collected data will be used etc. • Make sure the user understands and agrees • The user may leave whenever she/he wants • Confidentiality

  20. Types of measures • Experience-attitude • Performance • Cognitive

  21. Strengths: Utility: Can address most usability attributes Validity: User-centered; we ask for the user’s opinions Weaknesses: Validity/Objectivity: based on the user’s subjective judgement Experience-attitude

  22. Strengths: Utility: Can be used for most tasks and in different settings Cost-effective: Quite easy to create a list of activities Weaknesses: Validity/reliability: The user may choose a solution path you didn’t think of, but that nevertheless is satisfactory Validity(senitivity): Ceiling or flooring effects: the task is too easy or too difficult Performance: completeness

  23. Summary of measures ++ = very good; + = good; - = not so good; -- = poor

  24. Relation between data-collection techniques and measures ++ = very good; + = good; - = not so good; -- = poor

  25. Relation between data-collection techniques and measures Measure Practicle limitations Purpose of test Data-collection technique

More Related