DEMONSTRATING IMPACT: AN INTRODUCTION TO OUTCOMES-BASED EVALUATION (OBE)

1. DEMONSTRATING IMPACT: AN INTRODUCTION TO OUTCOMES-BASED EVALUATION (OBE) Jennifer Peters, Grants Administrator, Library Development

2. Goals for This Session • What is OBE • Why evaluate? • What is an outcome? • How We Use OBE • Logic Model • Why We Use OBE • Accountability to funder • Value to YOU • Resources for Further Research

3. Why Evaluate? • “Libraries Change Lives”– do they? • Answer questions: • Are we achieving what we hope to achieve? • Are we being efficient? • Are we being effective? • Does this program work? Would another program work better? • Is this program a model for other libraries?

4. What is Outcomes-Based Evaluation? • Tool for program design • Tool for measuring your program’s effectiveness • Tool for planning future programs • Based on models developed in social service agencies and United Ways • Extension of the data gathering you already do • Focused not on what goes into your program but what comes out of it • Provides deeper understanding of your program’s value to the community

5. What is an Outcome? • “Benefits or changes for individuals or populations during or after participating in program activities.” (IMLS) • “Beneficial changes for program participants that include changes in their skills, knowledge, behavior, attitude, status or life condition.” (Texas State Library and Archives Comm.) • “A target audience’s changed or improved skills, attitude, knowledge, behaviors, status, or life condition brought about your project.” (Washington State Library) • “Outcomes occur as a direct result of an action—such as training, services, or teaching.” (Boulmetis and Dutwin) • “Changes in adaptive behavior and role status that are logical consequences of the services.” (Schalock)

6. OUTCOME = IMPACT • How did your program impact the lives, skills, abilities, knowledge, behaviors of the person or group who attended your program? How was that person or group changed? • How does your library impact the community?

7. Outcomes • Not how many worms the mother bird fed her baby, but how well the fledgling flew • Not how many people attended the smoking-cessation class, but how many people actually quit smoking

8. How to Implement Outcomes-Based Evaluation • Think “big picture” about why you do programming • Think “big picture” about your program • Why are we doing this program? • What community need will this program meet? • Who do we want this program to reach? • When do we want to do this program? • How do we want this program to impact our audience?

9. Logic Model • Describes a library program • A planning tool to pull all the pieces of your program together • Helpful because it gives you the opportunity to think the program through from start to finish, with a focus on the impact of your program, the identified need it will meet, and its value to the community (AKA “outcomes”)

10. Wedding Logic Model **Borrowed from a presentation by Sue Ellen Schumacher, American Association of Grant Professionals, Nov. 2009.

11. Program Logic Model

12. Community Need

13. Community Need

14. Inputs

15. Inputs

16. Activities

17. Activities

18. Outputs

19. Outputs

20. Examples of Outputs Reference questions answered Persons attending ESL classes Books cataloged Web pages accessed  Items requested through inter-library loan Summer reading participants Newspapers digitized Computers purchased Library cards issued Classes taught

21. Outcomes

22. Creating Your Outcomes • Based on your program plan and community need, you should be able to easily identify 1-3 outcomes for your target audience • What is the need? • How do you want people to change? • What do you want the impact to be? • Think about short-term outcomes/intermediate/long-term outcomes • You develop them yourself—based on what you want the program to achieve, and what you want to communicate about the program

23. Integrating Outcomes into Your Logic Model

24. Integrating Outcomes into Your Logic Model

25. Integrating Outcomes into Your Logic Model

26. Integrating Outcomes into Your Logic Model

27. Integrating Outcomes into Your Logic Model

28. Integrating Outcomes into Your Logic Model

29. Integrating Outcomes into Your Logic Model

30. Integrating Outcomes into Your Logic Model

31. Integrating Outcomes into Your Logic Model

32. Integrating Outcomes into Your Logic Model

33. Integrating Outcomes into Your Logic Model

34. Integrating Outcomes into Your Logic Model

35. Examples of Program Outcomes • Students gain an understanding of basic navigation and how to locate a web page. • Library patrons become aware of the resources available on online databases. • Students show increased critical thinking skills. • ESOL learners gain ability to read directions and instructions in English. • Students demonstrate an appreciation of regional authors. •  Participants understand the value of voting in local, state and national elections. • Program attendees recognize and are sensitive to cultural differences in the clinical setting. • Pre-school children recognize that the value of books for learning and for fun.

36. Short, Intermediate, Long-Term Outcomes

37. Why Use Outcomes-Based Evaluation? Most funding agencies, including foundations, United Ways, and federal agencies, now require an outcomes-based evaluation model • For funders: • Shows a Return on Investment • Provides measures that they can, in turn, report on and aggregate to show the value of their investment in library systems • Provides evidence of “best practices” that can be shared with other libraries

38. Why Use Outcomes-Based Evaluation? • For YOU • Provides an opportunity to think critically about why you make the programming choices that you do • Provides deeper understanding of your programming for management, staff, volunteers • A well-planned logic model means a strong, cohesive program • Gives you the opportunity to evaluate your program internally and plan for the future • What worked? • What can we do better next time? • Gives you the opportunity to “tell your story.” Don’t forget to collect a “success story.” • Media • Local officials • Donors

39. Other Resources • Boulmetis, John and Phyllis Dutwin. The ABCS of Evaluation: Timeless Techniques for Program and Project Managers. Jossey-Bass, 2005. • Measuring Program Outcomes: A Practical Approach. United Way of America, 1996. • Outcomes Framework Browser. http://portal.whatworks.org/programs.aspx • “Outcomes: Libraries Change Lives – Oh yeah? Prove it.” IMLS presentation to Public Libraries Association, 2002. www.imls.gov/ppt/PLA-02-2OBE.pps • “Perspectives on Outcome Based Evaluation for Libraries and Museums,” IMLS. http://www.imls.gov/pdf/pubobe.pdf • Rubin, Rhea Joyce. Demonstrating Results: Using Outcome Measurement in Your Library. ALA, 2006. • Shaping Outcomes. Online course offered by Indiana University. http://www.shapingoutcomes.org/ • Texas State Library and Archives Commission. Outcome Measures. http://www.tsl.state.tx.us/outcomes/

40. Questions? • Contact Jennifer Peters, Grants Administrator, Library Development Division, Texas State Library and Archives Commission • jpeters@tsl.state.tx.us • 512-463-5527