Download
1 / 26
260 likes | 402 Views
Low-Cost Rapid Usability Testing: Is it Worth the Effort?. Tristin Baylis, MSc Senior Technical Consultant. Overview . Background What is Usability Testing Cost-benefit Analysis Results Questions. Faculty/Presenter Disclosure. Presenter: Tristin Baylis
E N D
Low-Cost Rapid Usability Testing: Is it Worth the Effort? Tristin Baylis, MSc Senior Technical Consultant
Overview • Background • What is Usability Testing • Cost-benefit Analysis • Results • Questions
Faculty/Presenter Disclosure • Presenter: Tristin Baylis • Relationships with commercial interests: • Tristin Baylis is a Senior Technical Consultant with CGI • This Study was conducted with the support of Andre Kushniruk – University of Victoria • Permission was granted by the BC Ministry of Health
Usability Testing Usability testing is a branch of usability engineering that focuses on analyzing and improving user interactions with computer systems.
Low-cost Rapid Usability Testing “Low-Cost Rapid Usability Testing” is a form of usability testing that removes the need for the expensive test lab A portable laboratory is created that may involve as little equipment as a laptop, a video camera and screen recording software taken into a real-world setting where testing is conducted. Low-Cost Rapid Usability Testing also implements a simplified approach to analysis of video data obtained from testing sessions.
Low-cost Rapid Usability Testing Low-cost rapid usability testing requires a small amount of physical materials. The basic testing setup requires a Laptop to run the software and equipment to record the users actions.The costs of all materials were tracked.
Cost-Benefit Analysis 1. Conducting Low Cost Rapid Usability Testing 2. Analysis of Costs 3. Cost-Benefit Analysis
BC Ministry of Health CDM Toolkit Chronic Disease Management (CDM) Toolkit: A secure web application that was developed as an information management and technology decision support tool to be used to support chronic disease management.
Conducting Low Cost Rapid Usability Testing Low-cost Rapid Usability Test Procedure: • Pre-Test Setup: Test equipment was setup, questionnaires prepared and testing data set to a consistent start point. • Overview: Before starting testing subjects were given an overview of testing process and asked to sign a consent form. • Performs Test Scenarios: 8 subjects were asked to complete 2 predefined test scenarios using the CDM Toolkit (Computer screens , audio and external interactions were recorded). • Post –Task Interview: When the subjects were done testing they were then asked several open ended questions while still being recorded. • Questionnaire: The subjects were then asked to complete a questionnaire about there overall impression of the system and the issues they encountered. • Analysis: The data collected (video recordings, screen capture and questionnaires) was then analyzed to determine application errors that were encountered and the associated severity of each error.
Cost-benefit Analysis The cost-benefit analysis was completed by comparing the cost of performing Low-Cost Rapid Usability Testing using 3 different scenarios: • Direct Measurable Savings • Cost of Errors related to when resolved in the SDLC • Cost of Medical Error
Analysis of Costs The first step of completing the cost-benefit analysis was to determine the total cost of the Usability testing which came to $8,362.91
Cost-benefit analysis As indicated in the table below a cost savings was found in all 3 cost-benefit cases
Discussion: Cost-Benefit Analysis In all cases a positive cost-benefit was found with savings attributed to the implementation low-cost rapid usability testing. The average cost savings was $15,774.45, which meant a total percent savings of 59.9% compared to the impact of errors going undetected and potentially causing a technology-induced error. It should be noted that the most conservative case was used in all cost-benefit analysis. If the extreme case was considered (i.e. Each Medical Error costing $600,000 ) a total cost savings of $1,791,637.09 could possibly be achieved.
Conclusion Early detection of errors (i.e. prior to production release) will allow organizations to achieve a 36.5% to 78.5% cost saving compared to the impact of errors going undetected and causing a medical error. Overall, Low-Cost Rapid Usability Testing was found to be a cost effective testing technique that can be implemented in conjunction with other testing techniques (e.g. unit testing, black box testing, white box testing, clinical simulations) in a cost effective manner to develop health information systems which will have a lower incidence of technology-induced errors.
References BC CDM Toolkit Documentation and Guides. Retrieved March 19, 2011 from, http://www.health.gov.bc.ca/access/guides/index.html BC Medical Services Commission Payment Schedule. Retrieved August 25, 2010 from, http://www.health.gov.bc.ca/msp/infoprac/physbilling/payschedule/index.html BC Ministry of Health Services Drug Data Files. Retrieved August 25, 2010 from, http://www.health.gov.bc.ca/pharmacare/outgoing/index.html Jaspers MW. (2009). A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. May; 78(5):340-53. Jeffries, R., Miller, J. R., Wharton, C., Uyeda, K.M. (1991). User interface evaluation in the real world: A comparison of four techniques. Proc. ACM CHI’91 Conf. (New Orleans, LA, April 28 May 2), 119,124. Karat, C., Campbell, R., Fiegel, T. (1992). Comparison of empirical testing and walkthrough methods in user interface evaluation. In Proceedings of CHI'92 (Monterey, California, May 3- 7, 1992), ACM, New York, 397-404. Karat, C. (1993).Usability Engineering in Dollars and Cents, IEEE Software, May 1993, pp. 88-89. Kushniruk, A. W., & Patel, V. L. (2004). Cognitive and usability engineering methods for the evaluation of clinical information systems. Journal of Biomedical Informatics, 37(1), 56-76. Kushniruk, A. W., Borycki, E. M.,(2006). Low-Cost Rapid Usability Engineering: Designing and Customizing Usable Healthcare Information Systems. Healthcare Quartely, 9(4), 98-100,102. McConnell, S. (2004). Code Complete (2nd ed.). Microsoft Press. pp. 960. Nielsen, J. (1993). Usability engineering. Boston: Academic Press, Inc. Neilson J. (1994). Usability Inspection Methods. New York: Wiley. Nielsen, J. (1994).Using discount usability engineering to penetrate the intimidation barrier, in R.G. Bias & D.J. Mayhew (eds), Cost-Justifying Usability, academic Press.
