Measuring Your Impact: Using Evaluation for Library Advocacy

1. Measuring Your Impact: Using Evaluation for Library Advocacy Canada Health Libraries Association May 31, 2009

2. Instructors Barb Jones Advocacy/Missouri Liaison jonesbarb@health.missouri.edu and Betsy Kelly Evaluation and Assessment Liaison kellyb@wusm.wustl.edu

3. Today’s plan • Agenda • Resources/Packet • Exercises • Housekeeping

4. Workshop objectives By the end of the workshop participants will: • Understand the library’s value in terms of the mission of the larger organization • Describe some tools used to assess the library, its users and stakeholders

5. Workshop objectives Identify the tools and methods used for data collection and analysis Design an evaluation plan for a service in a library Communicate evaluation results

6. What this workshop is not… • Custom-tailored • A quick fix • One size fits all

7. Demonstrating the library’s value

8. Why demonstrate value? • To show the impact of the library on the larger organization’s mission and goals • To show accountability for your resources; the library’s contribution to the bottom line • To demonstrate that the library provides value-added services • To use as an advocacy and marketing tool

9. How & what to evaluate • What you decide to evaluate depends on: • What you need or want to know • What your users feel is important • What certain stakeholders want to have evaluated • Use “smallball” evaluation • Base hits or home runs?

10. Means, not ends “Libraries are not ends in themselves, and they should not be supported because they have intrinsic value.” Plutchak TS. Means, not ends. J Med Libr Assoc 2004 Jul;92(3):294.

11. Library service is value-added • Cost savings • Institutional & Personal • Time savings • Travel, discovery, skills acquisition, opportunity for other uses of time • Access • Opportunity to use/apply information • Impact on clinical, research, safety, practice

12. The bottom line • You are either generating revenue, or supporting those who do OR … • You are helping to control operating expenses or supporting those who do OR … • You are creating expenses that add recognized value OR … • You are creating expenses that must be controlled or eliminated to reduce overhead Will Welton, PHD, Director, MHA Program, University of Washington

13. The big question! What is the bottom line and what is the library doing to support it?

14. What you need is a plan • Know the vision • Look at the environment • Create the plan • Collect and analyze the data • Communicate the value

15. The vision

16. Know what you want to do and why • Align the library’s mission & goals with the organization’s mission & goals • The library’s value = library’s contribution to achieving organization’s mission & goals

17. Royal Victoria Mission • Offer our pediatric and adult patients as well as their families compassionate exemplary care, with a specific commitment to treating complex cases; • Extend the limits of health knowledge through research and integrate this new knowledge to our clinical and teaching practices; • Provide outstanding health science education to healthcare providers, administrators and the community; and • Assess and develop new technologies and procedures to prevent, assess and manage medical conditions.

18. The environment

19. Why look at the environment? • Understand needs, desires and problems in context • Validate assumptions about your contributions and services • Provide a baseline for future evaluation • Help to develop the blueprint to plan and evaluate your contributions and services

20. Components of environment • The organization • Your library • Clients/users • Stakeholders • The community

21. Assessing the environment • User/stakeholder input • Surveys, Focus groups, Interviews • Unsolicited Feedback • Observation • Assessment by walking around

22. Assessing the environment • Library statistics and records • Collection use • Librarians work including reference, teaching, etc • Get out of the library and talk to people • Understand the role of information in their work • Learn how information is acquired, applied • Discover information problems they are trying to solve

23. SWOT analysis Weaknesses • Internal • Statements about what is lacking in your library Strengths • Internal • Positive statements about your library Opportunities • External • What do clients want you to do that you are not doing? Threats • External • Factors that can adversely impact your library’s goals

24. Plan Backward, Implement Forward

25. Where to go & how to get there Goals Outcomes Activities and outputs Resources

26. Goals are based on… • The library’s contribution to the organization’s mission and goals • Environment • SWOT analysis threats and weaknesses as well as the strengths and opportunities • Other assessments

27. Institution/Library mission Offer … compassionate exemplary care, with a specific commitment to treating complex cases Extend the limits of health knowledge through research and integrate this new knowledge to our clinical and teaching practices Provide outstanding health science education Assess and develop new technologies and procedures to prevent, assess and manage medical conditions Provide access to knowledge based health care information Committed to knowledge based health care practice and life-long learning Foster excellence in life sciences education Use the best of traditional and innovative technologies to facilitate information transfer

28. Goals/Objectives • Statement(s) of Purpose – why we do what we do and for whom • Mission: Committed to knowledge based health care practice and life-long learning • Goal: • Promote user self sufficiency • Provide physical and virtual learning environments

29. Outcomes • Outcomes are changes in attitude, behavior, skills, knowledge or situation • Short, medium and/or long term • Intended or unintended • Positive as well as negative • S.M.A.R.T. • Achieving outcomes impacts meeting goals!

30. SMART Outcomes • Specific: one or more web pages on the library website • Measurable: done or not; do staff access resources • Action-oriented: created and posted; staff access resources • Realistic: it is possible within the timeframe • Timed: “when” is articulated

31. Example Goal: Promote user self-sufficiency • Intermediate Outcome: Students will have skills to find information needed • Long Term Outcome: McGill graduates use evidence based medicine in their health care practice

32. Activities and outputs • Activities • What will you do? • Outputs • How many did you do? • How many attended? • How many were distributed? • How many times was it used?

33. Resources • What you have • Income • Equipment • Collection • Etc. • What you need • Operating expenses (e.g., personnel, acquisitions, maintenance, etc.) • Funds for new initiatives or services • Etc.

34. A POP QUIZ!

35. POP QUIZ: What is a logic model? • A beautiful person who thinks deep thoughts

36. POP QUIZ: What is a logic model? • A set of plastic pieces to assemble step by step

37. Activities/Outputs Resources Outcomes If we get these resources… Conduct these activities and deliver these products We will accomplish these outcomes POP QUIZ: What is a logic model? • A planning tool that links activities to results

38. The logic model • Helps organize your thoughts – both before and during a project/program • Provides a framework for planning and evaluating programs • Clarifies intended outcomes • Acts as a communication tool • Good source for more information: W.K. Kellogg Foundation Logic Model Development Guide (see class bibliography for link)

39. The model may change over time

40. Logic model worksheet Goal:_____________________________________________

41. Sample logic model • Goal: • Promote user self sufficiency

42. Logic model towards this goal

43. Logic model group activity • Choose a goal • Identify outcome(s) • List • activities • outputs • resources

44. Sample goals • Use one of your own goals • Or use one of ours: • Promote user self sufficiency • Provide physical and virtual learning environments

45. Wake up! Lunch time!

46. The evaluation plan

47. The evaluation plan • Builds on the logic model • Evaluates success • Assesses value

48. Consider the purpose • Who is your audience? This may be different than the beneficiaries of your services • Your users • Administrators • Fund raisers • How will the information be used? • Financial savings or justification • Intangible or non-monetary value of program benefits to community • Marketing and advocating for the library

49. What do you want to know? Have you achieved your stated outcome? “Students will have skills to find evidence based information needed for informed health care practice”

50. Pieces of the evaluation plan • Indicators • How will you know you have achieved the outcomes? • Data • Sources • Methods of collection • Frequency of collection • Resources • Expertise or tools needed to collect and analyze data