Topic 6: Usability Evaluation of IA Applications and Mechanisms

1. Topic 6:Usability Evaluation of IA Applications and Mechanisms Azene Zenebe, Ph.D. Bin Mai, Ph.D.

2. Presentation Outline • Introduction • Usability of IA applications and mechanisms - Reviewed • Usability Evaluation: What, When and Why • Usability Specification for Evaluation • Usability Evaluation Methods • Analytical methods • Empirical methods • Case Study • Summary

3. Learning Objectives and Outcomes • After completing this module, you should be able to: • Describe the factors that affect usability of security systems • Describe the importance of evaluation of usability security systems • Prepare usability specification for evaluation

4. Learning Objectives and Outcomes (Continued) • perform usability evaluation or testing of a security system using an analytical method such as expert inspection • Perform usability evaluation or testing of a security system using an empirical method such as a field study or lab testing • Report results of usability evaluation as well as describe how the results can be used to make improvement

5. Introduction • Usability of IA application and mechanism - Reviewed • Usability refers to the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of user - ISO 9241-11

6. Multi-dimensionality of Usability • Ease of learning • Efficiency of use • Memorability • Effectiveness • Error frequency and severity • Subjective satisfaction

7. “Usable” Security Systems • can easily and quickly learn a security system that they have never seen to accomplish basic tasks • can remember enough to use them later without major cost • are able to effectively perform and successfully complete security tasks supported by them • cannot make sever and frequent errors • are satisfied with the interface and functions of the systems

8. Framework for studying usability of security systems • four principal components in a human-machine system • TOOL • USER • TASK • ENVIRONMENT

9. Four groups of people involved in Security systems • Definers provide the policies, guidelines, and standards • Builders are the real techies, who create and install security solutions • Administrators operate and administer the security tools • End-users include home users and employees who are novice to CISS

10. Usability Evaluation: What, When and Why • Usability evaluation: whether a security system is usable for the users • Goal of usability evaluation: identify and correct flaws associated with ease of use of a security system • Performed during design and testing (or post-implementation) phases • Evaluation is iterative – an ongoing process

11. Usability Specification for Evaluation • Usability specifications are statements of required usability characteristics that are precise and testable • Task analysis provides a more precise specification of what users are expected to do in order to accomplish a task successfully

12. A sample usability specification - authenticity of a website

13. Usability Evaluation Methods • Analytical Methods - conduct analysis of a system’s features with the respect to their impacts for use • Empirical Methods – collect and use data from a system’s users. It is also referred as user-based testing

14. Analytical Methods • Expert’s knowledge stated as a heuristic rules • Ten Usability Heuristics by Jakob Nielsen • Shneiderman’s 8 Golden Rules of Interface Design

15. Empirical Methods • What usability evaluators want to know is what happens when users use the system • Different techniques are • Field studies • Usability Testing in a laboratory • Controlled Experiments

16. Quick Quiz • What are the main advantages and disadvantages for analytical methods and empirical methods? • Come up with two sample scenarios in IA field where you think analytical methods should be preferred, and two other scenarios where you think empirical methods should be preferred

17. Steps for usability testing • Identify and profile the representative users • Select the setting • Decide what tasks users should perform • Decide how and what types of data to collect • Perform necessary activities before test session • Perform necessary activities during test session • Perform necessary activities after test session

18. Usability Testing in a Laboratory • Validity concerns are associated with the following questions for lab based testing • Is the prototype system used in the testing missing any important features • Are test participants really the kind of users who will use the system • Will actual users do tasks like these participants • Will actual users be more distracted in their offices

19. Using the Results of Usability Testing • Results need to be looked at and actionable information regarding usability problems and issues should be made for design teams • Provide recommendations to address the identified problems

20. Automated Usability Testing Tools • A List of 24 Web Site Usability Testing Tools • http://www.usefulusability.com/24-usability-testing-tools/ • UMD list of usability testing tools • http://otal.umd.edu/guse/testing.html#sect3a • Jay Forbes’ presentation about usability testing tools • http://www.gslis.utexas.edu/~l385t6rb/auto_tools.pdf

21. Quick Quiz • Suppose you are testing the usability of an IDS your company decided to implement. • What will be the setting of the testing? • Who will be the representative users? • What type of data should you collect? Justify your answers.

22. Quick Quiz • Among IT managers, business managers, usability specialists, or general public, who do you think are the main users for automated usability testing tools? Why? •  What aspects of a usability study do you believe can never be automated? Why?

23. Case • Perspectives: Usability Evaluation • Perspectives is a new approach to help clients securely identify Internet servers in order to avoid "man-in-the-middle" attacks • works with Firefox 3 extension • Demo • Mission of the Perspectives • detect whether a self-signed certificate is valid • detect the fake security certificate attack and will warn you

24. Usability Evaluation Design • User Population • Potential Users: Novice, Intermediate and Expert in Security and IT • Targeted Users: Subset of the Potential Users • Context of Uses • Using the Internet • Home, free WiFi sites, and/or work • Quite or Not Quite environment • Tasks: Banking, Shopping, etc.

25. Usability Evaluation Design • Perspectives: evaluating the authenticity of a public key based on accompanying signatures and making use of a Browser’s built-in mechanisms for such evaluation • Requirements gathering • Develop usability specification • Usability Evaluation • Using Inspection • Using Empirical

26. Summary • From this module, reader should take away the following: • Usability is a combination of factors • Usability requires that users understands the organization policy and rules • There exist frameworks that guide the usability evaluation • For different stakeholders, the goals of usability differ

27. Summary (continued) • From this module, reader should take away the following: • Usability specification is required for usability evaluation • There are two categories of usability evaluation methods • There existing some tools that automate usability testing

28. Discussion Topics • What are the advantages and disadvantages of Inspection method? • What are the advantages and disadvantages of Empirical method? • Compare and contrast the different methods of data collection. Describe the advantages and disadvantages of these methods.

29. Discussion Topics • How useful are these Heuristics for security systems? Which of the two is more relevant to security systems? Are these methods security systems dependent? • Is there a heuristics for security system interface design? Is there a methodology?

30. Discussion Topics • Describe and discuss scenarios where a system’s usability is important to one type of users, while not so important to another type • What are your opinions regarding the ideas that, as described by Jay Forbe, “automated usability testing is too good to be true”?

31. Project Ideas • Suppose your friend Joe opened an E-bay store online to sell his comic book collections, what data do you collect to evaluate his website’s usability? • Suppose a university Registrar Office hires you to evaluate the usability of its online registration system. What data would you collect?

32. Project Ideas • Prepare a sample usability specification built to track usability of a scenario for setting a firewall in Windows XP. • Develop a usability evaluation desing to track usability of an IDS (Intrusion Detection System) • Design a usability evaluation study for the latest release of PGP.

33. References 1. Braz, C. and Robert, J.-M. Security and usability: the case of the user authentication methods. In Proceedings of the 18th International Conferenceof the Association Francophone d'Interaction Homme-Machine ACM, Montreal, Canada 2006 199-203 2. Garfinkel, S.L. Design Principles and Patterns for Computer Systems That Are Simultaneously Secure and Usable Department of Electrical Engineering and Computer Science, MASSACHUSETTS INSTITUTE OF TECHNOLOGY, Boston, 2005, 470. 3. Hoonakker, P., Bornoe, N. and Carayon, P., Password Authentication from a Human Factors Perspective: Results of a Survey among End-Users. In 3rd Annual Meeting of the Human Factors and Ergonomics Society, (San Antonio, TX, 2009). 4. Josang, A., Alfayyadh, B., Grandison, T., Alzomai, M. and Mcnamara, J., Security usability principles for vulnerability analysis and risk assessment. in Twenty-Third Annual In Computer Security Applications Conference, (Miami Beach, Florida, 2007), 269-278. 5. Lazar, J. Web Usability: A User-Centered Design Approach. Pearson, Addison Wesley, Boston, 2006. 6. Nielsen, J. Usability Engineering. Morgan Kaufmann, San Francisco, 1994.

34. References 7. Rosson, M.B. and Carroll, J.M. Usability Engineering: Scenario-based development of human-computer interaction. Morgan Kaufmann, San Francisco, 2002. 8. Shackel, B. Usability - Context, Framework, Definition, Design and Evaluation. in Richardson, S. ed. Human Factors for Informatics Usability, Cambridge University Press, Cambridge, 1991. 9. Shneiderman, B. and Plaisant, C. Designing the User Interface. Addison-Wesley, Boston, 2005. 10. Weir, C.S., Douglasa, G., Carruthers, M. and Jacka, M. User perceptions of security, convenience and usability for ebanking authentication tokens. Computer & Security, 28 (1-2). 47-62. 11. Whitman, M.E. and Mattord, H.J. Management of Information Security. Course Technology, Thomson Learning, Inc., Canada, 2004. 12. Whitten, A. and Tygar, D., Why Johnny can't encrypt? In USENIX, (1999).