Lessons Learned: How to Survive the Development of Outcome Measures

1. Lessons Learned: How to Survive the Development of Outcome Measures Cherie McCraw, M.L.S., Library Consultant, Division of Library and Information Services Linda B. Schrader, Ph.D., Florida State University

2. Overview • Overview of the four-year implementation process • Development of an outcome evaluation system for the State Library of Florida • Examination of lessons learned and challenges encountered

3. History • 1993 Government Performance Results Act • Requires federal agencies to establish specific objective, quantifiable, and measurable performance goals for each program • 1996 reauthorization of Library Services and Technology Act (LSTA)

4. LSTA Grant Background • State-based program • Focuses on information access through technology and information empowerment through special services • Emphasizes public libraries, but encourages all types of libraries to apply • Emphasizes accountability and evaluation

5. Institute of Museum and Library Services (IMLS) • Turned to outcome-based evaluation for LSTA grants • United Way model • Looks at the effect of an institution's activities and services on the people it serves • Powerful tool for identifying successes and communicating their value

6. IMLS Plan • Five states were selected to implement outcomes evaluation of LSTA grants early and develop model programs • Florida • Minnesota • North Carolina • Maryland • Washington

7. Florida’s Four Year Implementation Plan • Development of outcome model and standardized outcomes • Marketing of outcomes evaluation process • Planning for staff training and technical assistance • Conducting outcome evaluations

8. Implementation Activities • Administrative and management tools • Senior level buy in • State Library and field training

9. State Library of Florida’s Outcomes Timeline 1998 IMLS selects states for pilot projects 1998 Consultant hired to help develop model 1998 State Library staff receive United Way Outcomes Evaluation training Oct 1998 Outcomes Evaluation concept presented at annual Public Library Directors’ Conference 1999 Standardized outcomes for 6 LSTA categories developed 1999 OPPAGA Review favorable toward outcome concept Oct 1999 Outcomes Evaluation program presented at annual Public Library Directors’ Conference Nov 1999 Born to Read Capacity Building Workshop (outcome plans developed) July 1999 Florida’s “LSTA Outcome Evaluation Plan” published Fall 1999 Workshops on outcome evaluation presented Jan 2000 LSTA Grants workshop featured new guidelines Mar 2000 LSTA applications required outcome plan (not scored) Sep 2000 “Workbook: Outcome Measurement of Library Programs” published Oct 2000 Panel discussion of outcomes at annual Public Library Director’s Conference Nov 2000 LSTA Capacity Building Workshop (outcome plans refined) Oct 2000 Outcomes based LSTA grants began 2000 LSTA categories changed from 6 to 2/new standardized outcomes created Mar 2001 LSTA applications required outcome plan (scored) Jul 2001 Outcomes based evaluation workshops conducted Nov 2001 LSTA Capacity Building Workshop (evaluation strategies and tools) Nov 2001 LSTA annual reports reflected outcomes evaluation Dec 2001 LSTA Grants workshop included session on outcomes

10. Development of Outcome Model • Hired consultant • Adapted United Way Outcome Model • Identified standardized state outcomes • Revised rules, guidelines and forms

11. Outcome Model Components • Inputs • Activities • Outputs • Outcomes • Evaluation Indicators • Evaluation Sources/Methods

13. Standardized State Outcomes • Characteristics • Broad scope, relevant • Easily understandable • Focus on skills/behaviors • Can include quantitative and qualitative indicators • Tailored to local programs

14. Access for persons having difficulty using libraries Persons having difficulty using library services use services or information that were not previously available Library Technology Connectivity and Services Public uses technology to get information Public learns to use technology Standardized State Outcomes

15. Golden Gateways Library Family Learning Centers Small Business Information Resource Center REACH: REmote ACcess to the Homebound Parents Plus Born to Read Juniors to Seniors: Hillsborough Remembers Library Elderly Outreach (LEO) Project Access for persons having difficulty using libraries

16. Development of the Everglades Information Center Electronic Library: Community Training and Outreach Seniors Connect @ Jacksonville Public Library Mobile Training Lab Osceola Internet Improvement Library Automation: Moving Legal Information Toward the Public Mi Servicio de Biblioteca Library Technology Connectivity and Services

19. Marketing the Outcome Process • State leadership vanguard for change • Paradigm shift in what gets measured • Senior level buy in of new evaluation process • Continuous communication throughout the four year period • Ongoing support

20. Staff Training • Developmental approach to training • Technical assistance provided by regional library consortiums and state library consultants • Statewide conferences

21. Conducting Outcome Evaluations • Phased in requirements for outcome evaluations in grant process • Continued training and support to field staff • Initial review of annual reports

22. Lessons Learned • State Library assumed leadership role • Implementation phased in over time • Standardized the outcomes • Measurement tools could be the same • Better communication between libraries and grant partners • Multiple training opportunities

23. Challenges • Resistance from State and field staff • Cannot be implemented quickly • Adapting local programs to the standardized outcomes • Tailoring existing data collection tools • Libraries and grant partners interpret outcomes evaluation differently • Different levels of knowledge about outcomes

24. Where Do We Go From Here? • Continue training field librarians • Promote/pilot Outcomes Model for other library programs • Develop new Long Range Plan using outcome measurement