Discovery Tool Evaluation: A Three-Pronged Approach

1. Discovery Tool Evaluation: A Three-Pronged Approach 9th Annual AMICAL Meeting and Conference Sharjah, UAE

2. Background • AUS implemented Encore Discovery in 2009 • Other products like Serials Solutions Summon and EBSCO’s EDS are actively marketing in the Middle East • Decision in 2010 to evaluate all products • Having Encore installed gave us opportunity to evaluate products against the same databases, and think about how an ILS-integrated discovery solution compares to a less integrated solution Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

3. Methodology • Setting up trials of Summon and EDS • Scheduling vendor presentations of Summon and EDS • Testing all 3 products for ability to produce useful, relevant and accessible results • Reviewing the features of each of the products • Performing usability testing with all products • Assessing the costs of implementing each product • Assessing the impact of implementing each product on other library systems and operations Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

4. Usability Testing Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

5. Informal Usability Testing • Think Aloud Protocol • 5 participants per product • Participants only told they would be testing a new discovery tool to find library resources • Participants used their own research topic to complete realistic tasks • Observers did not interact with participant except to give tasks and encourage thinking aloud Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

6. Testing Environment • 1 test administrator • 1 observer • Laptop with built in microphone for recording voice • CamtasiaStudio with Powerpoint plug-in to capture screen and voice • Browser of choice • Other software, websites as requested • Google and Microsoft Word for spelling Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

7. Tasks Based on common assignments requirements seen at the Research Help Desk, participants were asked to find: • an article • an article from a peer reviewed/scholarly/academic journal • a newspaper article • a reference book • a general collection book • a book published in 2008 or 2009 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

8. Observations • Limit / facet confusion • “Format” or “Content Type” vs “Location” • Books, e-books, e-journal • E-journal format was used to find articles • Limits reset for each new search • Click through to article • Expect to get article, but given a WebBridge or 360 Link page Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

9. System Usability Scale • Participants filled out System Usability Scale questionnaire for perceived ease-of-use score • 10 item Likert scale questionnaire used to determine user’s perception of the system for • Effectiveness • Efficiency • Satisfaction • Resulting scores range from 0 to 100 • Average SUS score across 500+ studies is 68 Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

10. System Usability Scores Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

11. Overall Usability • Summon • Pop-up abstract and number of results for limit in brackets were very popular • Encore • Worked very well for books • Article search interface (Synergy) not well understood or used • EDS • Limits not intuitively named, extra “Update” click to refine • Can set mutually exclusive limits giving 0 results • Preference to use web for finding newspaper articles Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

12. Relevancy and Functionality Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

13. Relevancy Test • Testers - AUS public service librarians • Searches - Actual searches done this year in Encore • Scoring - Scale 1 - 5 + comments • Scorecards - One scorecard for each search in each discovery tool.   Extensive comments • Discovery tools tested • Encore • Summon • EDS Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

14. Actual SearchesSelected from Google Analytics Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

15. Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

16. Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

17. Encore URLs • /iii/encore/search/C|Scarbon+fiber+reinforced+polymer|Orightresult|U1?lang=eng&suite=pearl • /iii/encore/search/C|Sglobalization|Orightresult|U1?lang=eng&suite=pearl Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

18. Searches • arranged+marriage • carbon+fiber+reinforced+polymer • Electronics+All-In-One+Desk+Reference+For+Dummies • electronics+for+dummies • globalization • jean+luc+godard • sex+education • uae+federal+laws • walt+disney • "Artificial+intelligence"+And+"Manufacturing" Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

19. Types of Results • Top 5 results • Number of: • Books • Articles • Other (e-books, e-journals, DVDs, e-resources, print journals, videos, etc) Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

20. Types of Results Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

21. Relevancy Assessment Overall reaction to top 5 results: 5 Excellent (just what I was looking for) 4 Very good (on target and useful) 3 Fair (acceptable) 2 Poor (not useful) 1 Bad (unrelated to topic) Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

22. Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

23. Results Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

24. Functionality Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

25. Functionality Conclusions Basically, all 3 systems offer similar user functionality – sorting, limiting, emailing and “did you mean” are available. There is no superior feature that would make one system preferable to another, just as there is no missing feature that would eliminate a system. Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

26. Impact Assessment Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

27. Pieces & Configurations Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

28. Functional Equalization • EDS Federated Search • EDS & Summon – Millennium Integration Pieces • III Mobile Interface (free if Airpac) • III Encore Harvesting • III Query API – in development Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

29. Costs • Sunk Costs to write off • Encore Server • WebBridge License • Ongoing Costs • Quotes for each component Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

30. Work Needed for any Change • Implementation • Profiling of resources and catalog records • Interface customization where possible • Changes to website • Changes to electronic resources • Staff training • Modify instruction materials and curriculum • Ongoing Work • Bibliographic Data Loads • Changes to Holdings Loads for EDS Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

31. Potential Lost Functionality • Tag Clouds & Authorities for related searching • Availability Facet • WebBridge/Pathfinder for Collection Maps • User tags • Relevancy modification • Millennium integration • Automatic update/syncing • Hold placement • Collection map links • My Lists • Saving RP Groups in My Millennium Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

32. Potential Added Functionality • Administrative interface for searching & user interface • Mobile Interface bundled in • Repository Integration bundled in • API’s bundled in • 1-Click in 360 Link Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

33. Final Scoring Discovery Tool Evaluation | American University of Sharjah | 6.04.2012

34. Thank you Thomas Hodge thodge@aus.edu Kathleen Botter kbotter@aus.edu Mary Ann Nash mnash@aus.edu