Putting Evidence-based Practice to Work: Usability Testing at Northwestern University Library

1. Putting Evidence-based Practice to Work: Usability Testing at Northwestern University Library Frank CervoneAUL for Information TechnologyMonday October 29, 2007Internet Librarian 2007

2. Overview • The problem of web site design • Using more formal methods • Formal methods in practice • Outcomes

3. The problems of web site design • Web development/HCI is an intricate mix of technology and design • While there are exceptions, the majority of librarians haven’t been trained in HCI • Gaps in understanding the significant differences between the on-line and in-person experience • Once a site is done, most people are not eager to have others suggest adjustments or changes, particularly to those things they “know” are right Liberally adapted from Problems with in-house website development by “The IT Guy”http://www.newfangled.com/contentmgr/showdetails.php/id/7441

4. Web site design and organizational responsiveness • The more an organization depends upon its publics for achieving its mission, the more it should employ dialogic features into its Web site design • The challenge is to move the tasks associated with Web site design and maintenance away from being a "B-list" job to being an imperative for the survival of highly stakeholder-dependent organizations Kent, M. L., Taylor, M., and White, W. J. (2003). The relationship between Web site design and organizational responsiveness to stakeholders. Public Relations Review, 29(1), pp. 63-77.

5. Which leads to … • Evidence-based information practice • “…an approach to information practice that promotes the collection, interpretation, and integration of valid, important, and applicable user-reported, librarian-observed, and research-derived evidence” Andrew BoothSenior Lecturer in Evidence-based Healthcare InformationSchool of Health and Related ResearchUniversity of Sheffield

6. What is evidence-based librarianship? • Data provides the primary evidence for making decisions • Not anecdotal stories • Not “common sense” • Evaluation occurs early in the process • How is this different from what often happens now? • Decisions are made based on • Beliefs of what is needed • Frequently, these beliefs are inherently biased • Assumptions, anecdotal evidence, and preferences • Evaluation, if it occurs, happens afterward

7. Evidence-based professional practice • Derived from the evidence-based model of medicine • Fundamental precepts • Study the phenomenon • Contrast results to other studies of the same or related phenomena • Combine results to better understand the phenomenon at hand

8. Evidence-based practice process Redefine the problem Define the problem Evaluate change Find the evidence Apply results of the evaluation Evaluate the evidence

9. Defining the problem • Very similar to concepts used in “user-centered design” • Five points • Yes, there’s an acronym - SPICE • SETTING • Where is this being used; what is the context? • POPULATION • Who are the (potential) users? • INTERVENTION • What is being done to/for them? • COMPARISON • What are the alternatives? • EVALUATION • What does success mean?

10. Levels of evidence • Systematic review of a wide-body of rigorous studies • Systematic review of a wide-body of less rigorous studies • Randomized controlled trials • Controlled-comparison studies • Cohort studies • Descriptive surveys • Case studies • Decision analysis • Qualitative research (focus groups, ethnography, historic, Delphi techniques, interviews) • Surveys, audits Adapted from Eldredge, J. (2000). Evidence-based librarianship: An overview. Bulletin of the Medical Library Association, 88(4). pp: 289-302. Online at http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=35250

11. Which brings us to the Northwestern story… • First usability test in 2001 • Focused in on “Electronic Resources” • A home-grown resource finder • Investigation and questions based on anecdotal evidence with ER interface • Learning experience • Provided model for organization • Whetted people’s appetite for usability testing

12. The second test – Searching in NUcat • Began summer 2002 • How are people using Voyager? • Are there discernable patterns? • Most importantly are there patterns of errors? • Can we do something to fix that? • What we did • Looked at searching patterns in our Voyager system • Used a more rigorous research approach • Based on grounded theory • Simple data mining techniques

13. And in the process… Searches by type

14. …dispelled some myths No hits within searches by type

15. Which lead to our tackling a bigger problem • Existing site did not work especially well for the Library's various user communities • Both students and faculty expressed frustration with the web site • Ability to navigate the site • Overall look and feel • Problem areas determined by gathering information in focus groups, online surveys, interaction with librarians and staff • Hours and locations • Collections • Electronic resources

16. Moving to a more evidence-based process • Restructuring of the web development process • Creation of a active web group • Library Web Advisory Group • Representation from all divisions of the Library • Reference/Instruction • Collection management • Bibliographers • Branch libraries • Technical Services • Library Public Relations • Digital Library Systems Development • Library Information Technology Support Services • Web Communications Group at University Relations

17. Induction process • Required reading list • Extensive list of supplemental readings • Required training • In usability • Conducting a usability test • Other “soft skills” • Evidence-based procedures • Required participation in design and testing • Usability test moderator • Usability test observer • Transcriber • Missionary/representative

18. Current results • Overall, site usability has improved • And we can provide it with statistical measures! • People find resources more quickly and easily • But there is still an unfortunate amount of confusion • Debates are less rancorous about how to proceed • We can always go back to the data • We can always go back to the studies • Easier to develop a strategy for incremental improvements over time • No longer locked into a tight academic schedule

19. Remaining issues • Jargon • I don’t know what • Metasearch • E-journals • Virtual reference • Electronic resources means! • Why should I go here? I can find everything in Google (or Google Scholar) • To some degree, this is true…

20. The big issue • Designing for a world where they don’t even come to the library web site • When the day comes that Google Scholar or Microsoft Academic Live do provide a “universal index” • Pluggable modules for the University portal

21. Thanks • Frank Cervone Assistant University Librarian for Information Technology Northwestern University Library 1970 Campus Drive Evanston, IL 60208 email: f-cervone@northwestern.edu AIM/YAHOO: hfcervone Skype: frank.cervone