1 / 16

Test and Evaluation of An Electronic Database Selection Expert system

Test and Evaluation of An Electronic Database Selection Expert system. Wei Ma (weima@uiuc.edu) Timothy W. Cole (t-cole3@uiuc.edu) University of Illinois at Urbana-Champaign. Overview. Development of the Smart Database Selector Usability Test - Objectives & Methodology Results:

pamelahammond
Download Presentation

Test and Evaluation of An Electronic Database Selection Expert system

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Test and Evaluation of An Electronic Database Selection Expert system Wei Ma (weima@uiuc.edu) Timothy W. Cole (t-cole3@uiuc.edu) University of Illinois at Urbana-Champaign

  2. Overview • Development of the Smart Database Selector • Usability Test - Objectives & Methodology • Results: • Analysis of Recall & Precision • Analysis of Failed Searches • User Satisfaction Measures • Problems of Focus Group Usability Testing • Conclusion & Ongoing Work 10th ACRL National Conference

  3. Development of the Smart Database Selector • Challenges: • Growing Number of Electronic Resources • Constant Change • Different Access Points • Heavy Demand • Users’ Difficulty in Selecting Databases • Objectives: • Computer-Assisted Technique for Guiding Users to Databases • Improve User Service & Facilitate Access 10th ACRL National Conference

  4. Interface as Tested 10th ACRL National Conference

  5. Keyword(s) search +/ Other Criteria Controlled vocabulary table Subject category term table Tables of db characteristics Recommended databases Search Form 1: Keyword search process: 10th ACRL National Conference

  6. Search Form 2: Search process for Browse List of Db Topics: Browse List of Db topics +/ Other Criteria Recommended databases Search Form 3:Search process for Select by Type of Information: Recommended databases Select by Type of Information Subject category term table Tables of db characteristics Tables of db characteristics 10th ACRL National Conference

  7. Test Objectives & Methodologies • Objectives: • Measure System Performance • Discover User Usage Patterns • Identify Ways to Improve System • Solicit User Reactions • Methodology: • Focus Group Usability Test • Transaction Log Analysis 10th ACRL National Conference

  8. Focus Group Population 10th ACRL National Conference

  9. Test Search Scenarios 10th ACRL National Conference

  10. Formulas Used to Calculate Recall, Precision, andStandard deviation 10th ACRL National Conference

  11. Results — Recall & Precision(Per Question, Library User Group) 10th ACRL National Conference

  12. Results — Failed Searches 10th ACRL National Conference

  13. Summary of Observations • Search Success Measures • Keyword Searches Preferred • Keyword Searches Much More Successful Against Controlled Vocabulary Term Lists • Both Complete & Sampled CVs Useful • Success Measures Varied by Question • Library Staff Recall & Precision Better • Doing More Than 1 Search Helped • Failed Searches: • Poor Word Choice Most Common Problem • Too Many Limits More Harm than Good 10th ACRL National Conference

  14. User Satisfaction Measures • 17 of 22 Agreed / Strongly Agreed that SDS Was “Useful in Recommending Database” (with 2 Remaining Neutral) • 20 of 22 Favored SDS as Compared to Current Library Gateway Menu System • 15 of 22 Found SDS “Interface Easy to Use” (with 3 Remaining Neutral) 10th ACRL National Conference

  15. Problems of Using Focus Groups for Usability Test • Focus Group: • Not as Familiar with the Assigned Topics • Play with the System Out of Curiosity • Different Motivation • Real Users: • More Knowledgeable About Topic Vocabulary • Strong Desire for Specific Information —More Interested in Finding Relevant Citations (Rather than Finding Relevant Databases) 10th ACRL National Conference

  16. Conclusion & Ongoing Work • Improvements Made in Interface Design • Now Offer 2 Options “Basic Search” & “Advanced Search” • Each Search Screen Has 2 Forms • Consolidated Optional Limit Criteria • Added Authority Control to Procedures for Assigning Subject Category Terms • Provided More on Screen Search Hints • 2nd Round of Usability Last Fall • Preliminary Analysis Supports Changes • Now Integrating Tool into Library’s Gateway 10th ACRL National Conference

More Related