Measuring Website Usability: Instrument Development, Validation, and Application

1. Measuring Website Usability: Instrument Development, Validation, and Application Big XII IS Research Symposium April 5, 2003 Younghwa “Gabe” Lee University of Colorado at Boulder leey@colorado.edu

2. Agenda • Background • Previous Studies • Research Objective • Research Design • Expected Contribution • Discussion

3. Background • Online business failures are increasing as customers turn away from unusable or unfriendly sites. ‘Build it and they will come’ mentality has led to the demise of e-commerce sites when sites are too late, too buggy, or too complex - Becker and Mottay, 2001 • In a poorly designed EC environment, users might be uncomfortable with the uncertainty and ambiguity caused by lack of interaction with websites. • Jahng et al, 2000 • The number of shoppers and total sales are still marginal, mainly because of poor interfaces - Jarvenpaa and Todd, 1997

4. Background • Building a usable website is important since website is the only source for online customers to touch, feel, search, communicate, and experience the products or services available at the online store • Usable websites • Build positive attitude (Singh and Dalal, 1999) • Increase stickiness (Rettie, 2001) • Increase revisit rates (Klein, 1998) • Increase online purchase (Palmer, 2002) • Increase performance (Nielsen, 2000) • Provide more satisfaction (Lund, 1999)

5. Website Usability Research Website Usability How to measure? How to verify? Instruments Development Design and Testing • HCI • Company-specific • Industry Gurus • Few IS researchers • HCI • MIS SIG HCI

6. Previous Studies • Measurement problems of website usability • No consensus on the definition and dimensions of website usability • A number of single-item constructs • Intuition and experience-based: Few efforts to develop measurement using scientific methods • HCI-oriented objective variables (error rate and download time) • No Investigation of the relationship between website usability constructs • No process model

7. Palmer (2002) • Download Delay • Navigability • Information Content • Interactivity • Responsiveness • Kim et al. (2002) • Firmness • Convenience • Delight • Agarwal and • Venkatesh (2002) • Content • Ease of Use • Promotion • Made-for-the-medium • Emotion • Zhang and • von Dran (2002) • Content • Enjoyment • Privacy • User empowerment • Visual appearance • Technical Support • Navigation • Credibility • Organization • McKinney et al. • (2002) • Access • Usability • Entertainment • Hyperlinks • Navigation • Interactivity

8. Download Delay Navigability Information Content Perceived Success Interactivity Responsiveness Download Delay Navigability Information Content Perceived Success Interactivity Responsiveness Palmer (2002)

9. Motivation of the Study • Current inconsistency and incompleteness among website usability measurement is the crucial problem of website usability studies. • There is very little in the way of concrete measurement that tells us how good a website really is. Current guidelines, methods, and metrics do help to design better websites, but there is room for improvement - Tarasewich (2000)

10. Research Objective • Develop measurement of website usability • 18 constructs and 62 instruments have been identified • Investigate the causal relationship between website usability constructs • Revealed Causal Mapping approach (Nelson et al., 2000) • Examine the effects of website usability constructs to multiple dependent variables • Satisfaction, purchase intention, revisit intention, actual purchase, affect, and loyalty • Investigate generalizability of the new measurement and identify different causal maps under different boundary conditions • Gender, Product, Industry and Culture

11. Research Design • Instruments Development • Literature review • Interviews with web usability experts • SUN, IBM, 37 Signals.com website designers • A Focus Group Study (IS-majored master-level subjects) • A Major survey to 400 Business undergraduate students • Exploratory Factor Analysis • Causal Relationship Between Website Usability Constructs • Interviewed with experts (n = 20) • Interviewed with experienced online customers (n = 40) • Data Analysis suggested by Nelson et al. (2000)

12. Research Design • Effects of Website Usability Factors to Diverse Dependent Variables • The effects of Price, Time, Scarcity, Convenience, Fun, Usefulness will be examined together • CFA and Path Analysis will perform • Data Analysis: LISREL • Boundary Conditions • Compare websites with different gender-focused • Compare websites with different types of products (Hedonic vs Utilitarian) • Compare websites with different cultures (U.S. vs Japan) • Compare websites with different stakeholders (Customers, Designers, and Managers) • Data Analysis: PLS

13. Simplicity Timeliness Scope Readability Scanability Consistency Relevancy Learnability Content Navigability Flexibility Credibility Privacy Tele- Presence Security Interactivity Community Reliability/ Accessibility Website Usability Constructs

14. Causal Relationship between Website Usability Constructs Simplicity Scanability Timeliness Scope Readability Content Consistency + Relevancy Learnability + Navigability + - - Flexibility Tele- Presence Credibility Privacy - + + Interactivity Community Reliability/ Accessibility Security

15. Simplicity Scanability Timeliness Scope Readability Content Consistency Relevancy + Learnability Navigability + + - - Flexibility Tele- Presence Credibility Privacy - + + Interactivity Community Reliability/ Accessibility Security Effects of Usability Constructs to Diverse Dependent Variables Actual Purchase Revisit Intention Purchase Intention Satisfaction Affect Loyalty Price Time Scarcity Convenience Fun Usefulness

16. Expected Contribution • Develop new measurement of website usability • provide a better means to evaluate website design quality • Identify causal relationship between website usability constructs • Provide a deeper understanding of how online customers build their usability perception • Examine effects of website design factors to diverse dependent variables • justify the investment on website usability • Investigate measurement’s generalizability and causal maps under different boundary conditions • help to perform future study using the measurement • help designers to allocate limited resources to the most important usability factors

17. Discussion

18. Discussion

19. Revealed Causal Mapping Repertory Grid Causal Relationship Identification Construct Identification Guided Interviews Open Interviews No requirement for multiple elements Multiple elements are required Identify constructs and causal Identify constructs using Relationship based on repeated Triadic Methods to identify Interviews for identifying Mid-theory exists No requirement for mid-theory

20. Discussion • Alternative ways of conducting the research • Process Model (e.g TAM) vs Multiple Dependent Variables • Boundary conditions valuable to be observed