The Use of Outcome Based Evaluation OBE to Assess Staff Learning Activities at the University of Maryland Libraries

1. University of Maryland Libraries The Use of Outcome Based Evaluation (OBE) to Assess Staff Learning Activities at the University of Maryland Libraries Irma F. Dillon, Manager MIS Maggie Saponaro, Manager Staff Learning and Development ARL Assessment Conference September 27, 2006 Charlottesville, VA

2. University of Maryland Libraries Overview Outcome Based Evaluation (OBE) UM Libraries Learning Curriculum (LC) OBE at UMD Advice and Lessons Learned

3. University of Maryland Libraries What is OBE? Systematic way to assess the extent to which a program has met its intended results. Examines the impacts, benefits and changes to participants as a result of the program(s) efforts during and/or after their participation.

4. University of Maryland Libraries What is OBE? Examines impacts, benefits, and changes in the: Short term Intermediate term Long term

5. University of Maryland Libraries OBE � Program Elements Inputs Activities Outputs Outcomes Outcome Targets Outcome Indicators

6. University of Maryland Libraries Why OBE at UMD Libraries? Funding agencies requesting expected outcomes in applications LC purpose fits OBE since its purpose is to provide needed skills Evaluating LC�s impact using OBE made sense

7. University of Maryland Libraries Learning Curriculum and  Staff Learning Activities Introduced in 2001 Ten modules � ranging from Customer Service through Library Skills Open to library faculty, staff, graduate assistants and students Taught by in-house staff, consultants, and subject experts 175 workshops through August 2006 Cumulative attendance 2618

8. University of Maryland Libraries LC Evaluation Activities  (2002 � Present) Computer training surveys Supervisory focus groups Writing workshop survey Long-term surveys Session-end evaluations Outcome based evaluation

9. University of Maryland Libraries OBE - Current Implementation (Using a modified logic model) Target Audience � All library faculty, staff and graduate assistants Outcomes � Staff able to successfully practice skills gained Immediate � participants benefit from attendance Intermediate � participants begin to comfortably use the information/techniques Long-term � participants continue to use skills learned and attend LC programs

10. University of Maryland Libraries OBE - Current Implementation Indicators � observable and reported evidence of change by participants Change could include skills, attitudes, behavior or condition Inputs � assembling trainers, developing and providing training sessions, publicity, etc. Outputs � number of participants, number of sessions held

11. University of Maryland Libraries OBE Survey Questions What knowledge/skills did you learn at this workshop? Have you used or applied the knowledge/skills from the workshop? Are there other skills you wished had been taught at this workshop? What other topics would you like to see Staff Learning and Development Offer?

12. University of Maryland Libraries Survey Says�

13. University of Maryland Libraries Survey Says� 67% of respondents have used skills taught 82% of respondents indicate needed skills taught Responses show: Most feel the workshops have made a positive impact on how they do their job Would recommend the sessions attended to others

14. University of Maryland Libraries Next Steps: Issues to Consider Are we asking the right questions to truly learn whether we are producing the desired outcomes? Do we need to review the time frame for sending out the long-term survey?

15. University of Maryland Libraries Next Steps: Issues to Consider Are we mining the data thoroughly enough and gaining enough insight to make improvements in the LC offerings? Are there other tools that we should consider using to gather information, e.g., focus groups, interviews, etc.

16. University of Maryland Libraries Lessons Learned Make the outcomes uncomplicated Review the data currently on hand Decide how much data is needed for analysis and decision-making Make it easy for the staff gathering the data

17. University of Maryland Libraries

18. University of Maryland Libraries The Use of Outcome Based Evaluation (OBE) to Assess Staff Learning Activities at the University of Maryland Libraries Irma F. Dillon Maggie Saponaro ifd@umd.edu msaponar@umd.edu