Best-Fit Evaluation Strategies: Are They Possible?

1. Best-Fit Evaluation Strategies: Are They Possible? John Carlo Bertot, John T. Snead, & Charles R. McClure Information Use Management and Policy Institute College of Information, Florida State University http://www.ii.fsu.edu

2. Introduction • Received a grant from the Institute of Museum and Library Services (IMLS) to develop an online instructional system to assist public libraries evaluate their services/resources • Evaluation Decision Management System (EDMS) • In year two of three year project • Why this project? • Little research provided comprehensive assistance in determining what specific evaluation strategies serve libraries best relative to specific library situational factors and contexts, data needs, and other considerations • With so many evaluation options available, there was a need to bridge information-need issues (i.e., situational factors, data needs, stakeholder questions, etc.) with evaluation approaches • Understanding information needs and linking these needs to evaluation approaches required evaluation strategies capable of providing data that library decision makers can use to address specific problems • Bridge practice an research

3. Best Fit Evaluation Approaches • The original intent of the project was to develop • Best-fit evaluation strategies which match the data needs of a library within a specific situational context to the evaluation approaches that are most appropriate to that particular situational context • Identify which evaluation strategies supply the best data within specific library organizational and situational contexts for use to provide the greatest impact to improve library services, or enable libraries to better advocate the value of libraries to their institutions or the communities they serve

4. Best-Fit Evaluation Strategy Considerations • What evaluation approaches are available • Which evaluation approaches might best meet a library’s data needs, either library developed or imposed by external funders/organizations/etc. • How to develop an overall evaluation plan that makes effective and efficient use of limited library resources • How to implement an evaluation strategy • How to use evaluation findings to advocate for local library support

5. Academic Literature Outcomes assessment Value demonstration ROI Outputs Service Quality Other Stand alone literature and approaches Bring these to the public library community The Original Approach

6. Different Perspectives • Stakeholder Perspective • Answer a range of questions asked by various stakeholders groups (e.g., library boards, county or city executives) regarding library services and resources • Make informed decisions regarding a library’s range or availability of services and resources • Data Perspective, provide data to • Answer a range of questions asked by various stakeholders groups (user-centered evaluation perspective) • Make informed decisions regarding a library’s range or availability of services and resources (library-centered evaluation perspective) • Demonstrate value and effectiveness of the library to the community that it serves (community-centered evaluation perspective) • Frame the perceptions of the library in the local political environment (political context-centered evaluation perspective) • Support the notion of the library as serving as a public good (customer-centered evaluation perspective)

7. Then the Reality of Public Libraries • Advocacy • Tell story • Value • Impact • Cuts across individual evaluation approaches • Integrated • Fits library’s situation

8. Development of Best-Fit Evaluation Strategies • Library decision makers are often faced with difficulties matching their data needs with the appropriate evaluation approaches • There are many different kinds of evaluation data that a library may need and evaluation approaches that a library might employ • As a result, many libraries struggle with the problem of choosing the best evaluation approaches to effectively and efficiently demonstrate the value they provide • And…they don’t necessarily consider “evaluation”

9. Some Issues and Findings • Have not quite abandoned the “logic model” approach to evaluation and “best fit”, but have • Modified thinking significantly • Changed how we make evaluation tools available • Less “evaluation like” • Focused more on advocacy and story telling -- with evaluation on the side • Though it does underlie the entire approach

10. Some Issues and Findings • In essence, developing a community of practice • Resources • Access to experts • Information commons for asking questions and sharing ideas, resources, and other material • Successes and failures • How to • Other

11. Some Issues and Findings • Public libraries don’t necessarily consider evaluation as part of their advocacy strategy (if they have one) • The link between evaluation, data, and advocacy is not always apparent • Once burned, twice shy • If have used evaluation strategies and data collection efforts, and not met with success, there is a reluctance to do so again • Waste of resources • Library situational factors (organizational, community, other) affect the successful use (or unsuccessful use) of leading evaluation approaches • Though libraries now talk about advocacy regularly -- the extent to which they are able and/or ready to engage in advocacy strategies is in question • Regardless of a linkage between evaluation and advocacy • The fear is that we may have to hide evaluation in an advocacy wrapper • These are two different things and require different skill sets • The ability of a library director to negotiate the political environment of the library may trump evaluation

12. Some Issues and Findings • Need to provide as many tools as possible to facilitate evaluation • If, then interfaces that enable EDMS to know something about libraries • NCES data (operating budgets, population served, FTEs, other) • Public Library Internet data (connectivity, bandwidth, workstations, services, other) • Prepackaged reports which incorporate data • Peer comparisons • Brochures • Presentations (Budget, building, other)

13. Moving Forward • Challenges remain • Biggest is making link between evaluation and advocacy • “Painless” evaluation that meets library’s situational factors • Redefining success of project

14. Conclusions • Some libraries do continue to engage in a wide range of evaluation efforts to assess services and resources provided to the communities libraries serve • The evaluation environment is increasingly complex, and requires knowledge of multiple evaluation frameworks, methodologies, data analysis techniques, and communication skills • The issue is not that libraries face a lack of available evaluation approaches • The issue is helping libraries select the approach or approaches that best meet advocacy needs of a library from the many evaluation techniques that exist • Making the evaluation-advocacy link

15. Thank You • Questions/Comments? • Contact information • John Bertot: jbertot@fsu.edu • Information Institute: http://www.ii.fsu.edu