1 / 13

ZLOT Prototype Assessment

ZLOT Prototype Assessment. John Carlo Bertot Associate Professor School of Information Studies Florida State University <jcbertot@lis.fsu.edu>. Overview of Assessment. Purpose Assess the extent to which the vendor prototypes met the functional requirements as determined by ZLOT efforts

coral
Download Presentation

ZLOT Prototype Assessment

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ZLOT Prototype Assessment John Carlo Bertot Associate Professor School of Information Studies Florida State University <jcbertot@lis.fsu.edu>

  2. Overview of Assessment • Purpose • Assess the extent to which the vendor prototypes met the functional requirements as determined by ZLOT efforts • Method • Assessment form • The technical, functional, and other specifications as identified in the Functional Requirements report; • The review of a number of ZLOT-related documents and material; • Discussions with the ZLOT study principal investigator; • Consultant assessment of integrated system vendor products, literature, and other material to the extent possible; • Preliminary tests with demonstration and implemented integrated search and retrieval systems deployed in various library settings; and • An initial review of the evaluation form with the ZLOT study team during which suggestions for changes to a draft form were suggested and incorporated to the extent possible. December 16, 2002

  3. Overview of Assessment • Method (cont’d) • Pre-tests, final • Prototype assessment • 3 assessments per prototype (1 per reviewer) • Developed search queries in advance (subject, author, key word, title) • Primary string with secondary if no results yielded • Searched individually • OPACs • Databases • Web resources • All, if available December 16, 2002

  4. Overview of Assessment • Method (cont’d) • Could not assess some functional requirements • Priority 1 Requirements • SRI 15 Collect usage statistics • SRI 17 Meet accessibility standards • Priority 2 Requirements • SRI 3 Allow libraries to offer patrons access to a statewide common LOT interface and/or a locally customized version of the common LOT interface • SRI 4 Link to local library web sites • SRI 5 Support local customization • SRI 6 Interact with profiles of participating libraries • SI 13 Allow customized presentation of resource collections or resources at the local library level based on user groups and other criteria • SI 14 Allow local libraries to select a group of catalogs for searches that will be the default set for searching December 16, 2002

  5. A Note on the Analysis • Developed two indexes to facilitate presentation of findings • The Functional Requirement Capability Index • Devised by summing all the points that a single functional requirement (and its subcomponents, if applicable) received from each evaluator for all prototypes and dividing that by the total number of points possible for the functional requirement. • Points for each functional requirement were assigned for the following values: Yes = 2 No = 0 Somewhat = 1 • This index indicates the overall extent to which the prototypes were able to meet the functional requirements as specified by the Functional Requirement report December 16, 2002

  6. A Note on Analysis • The Functional Requirement Ability Index • Devised by summing all the instances in which a prototype was fully able to meet the functional requirement (and its subcomponents, if applicable) • [an indication of “yes” on the assessment form by the evaluators] received from the evaluation and dividing that by the total number of points possible for the functional requirement. • In effect, this provides an indication of the full extent to which the prototypes were able to meet the functional requirements as specified by the Functional Requirement report December 16, 2002

  7. Key Findings Search Interface Functional Requirement Capability and Ability Index. (N=36) December 16, 2002

  8. Key Findings Retrieval Interface Functional Requirement Capability and Ability Index. (N=36) December 16, 2002

  9. Key Findings Search and Retrieval Interface Functional Requirement Capability and Ability Index (N=36) December 16, 2002

  10. Summary of Findings • 10 Priority 1 and 2 functional requirements met at the 50% level or better (using ability index) • Priority 1 • SI 1 Provide simple keyword search functionality • SI 2 Provide basic search functionality • SI 5 Allow concurrent searching of multiple online catalogs • SI 6 Allow concurrent searching of multiple TexShare databases • SI 10 Group online catalogs by geographic proximity • SI 12 Limit searches to local resources • SRI 1 Present a consistent user experience • SRI 2 Enable independence of the interface from resources and changes in resources • Priority 2 • SI 7 Allow concurrent searching of multiple LOT resource collection types • SI 11 Group online catalogs by type of library December 16, 2002

  11. Concluding Comments • Assessment of prototypes such as these requires an iterative learning strategy that enables evaluators to make changes to the assessment protocol, strategy, and other aspects to better reflect the operating environment in which the evaluation activities occur. • System variances, technology difficulties and variation, and a number of local circumstances influence the evaluation process. • While a standard, agreed upon assessment protocol and method were developed by the consultant and ZLOT study team, there are evaluator differences that influence the assessment of a particular functional requirement and/or prototype. • In most cases, two out of three assessments concurred; however, variations did enter into the overall evaluation. • More evaluator coordination and training in the areas of functional requirement terminology, definitions, and protocol use, to name a few, may help to minimized this. December 16, 2002

  12. Concluding Comments • There is a need to have vendors provide a uniform set of documentation that the evaluators can review prior to the prototype assessment. • This documentation should describe and provide examples of how the vendor’s system engages the specified functional requirements. • The documentation does not necessarily need to be extensive, but should provide at least a basic description of the prototype’s operation vis-à-vis the functional requirements. • This evaluation process unfolded in a very short period of time. • The extent to which the time frame affected the findings remains unclear. December 16, 2002

  13. Concluding Comments • The proof of concept determined that • Multiple digital resource search and retrieval tools are “real world” applications • Room for improvement, but key elements are in place • Some considerations • Vendors may actually be able to meet additional functional requirements now • Not clearly able to determine/test • A key aspect will be use and usage reporting • Unable to determine the extent to which vendors comply to various standards of use statistics • http://www.niso.org/emetrics • Moving forward may require various upgrades to Texas library systems December 16, 2002

More Related