LS 403 Evaluation of Information Services

1. LS 403Evaluation of Information Services Introduction Laura Saunders

2. Why Take this Class? … besides that it’s required. Laura Saunders

3. Why Take This Class • Take 3 minutes to answer these questions… • What do you expect/ hope to learn from this class? • What do you want to be able to do after this course? Why? • Why is this field (evaluation research) important? Or is it? Laura Saunders

4. Background • Edward Suchman observed that “one of the most appealing ideas of our century [the 20th century] is the notion that science can be put to work to provide solutions to social problems” • Substitute “social sciences” for “science” • Expand “social problems” to problems faced by organizations Laura Saunders

5. Background: Evaluation vs. Social Research • Evaluation Research • Uses many of the same methods/ practices • Typically takes place within an organization- with its own culture and politics • Requires understanding/ ability to work with groups, personalities, teams, etc. • Requires sensitivity to issues and concerns of various interests involved. Laura Saunders

6. What and Why • According to Peter Hernon • The purpose of evaluation is “to collect data useful for documenting the quality and effectiveness of existing programs and services, and to use that information for service improvement, demonstrating best practices, or providing evidence of accountability. • According to William Trochim • Evaluation is the systematic acquisition and assessment of information to provide useful feedback about some object Laura Saunders

7. Purpose and Potential • Evaluation compares “what is” with “what should (could) be”- element of judgment • Evaluation must have a purpose, rather than be an end in itself. • There is no need to evaluate without the potential for action. • Evaluation must go beyond description to understand relationships. • Joseph Matthews Laura Saunders

8. Contexts • Evaluation research may generate conflict • Often begins with problem identification • Tends to involve groups of people, personalities, etc. • Competing priorities. • Purpose is improvement- often results in change. Laura Saunders

9. What and Why • Unfortunately, according to James G. Neal, • “decisions are routinely not supported by the evidence of well-designed investigations,” • “research in the field is poorly communicated, understood and applied.” • Neal, J.G. (2006). “The research and development imperative in the academic library: Path to the future.” portal: Libraries and the Academy, 6: 1. Laura Saunders

10. Current Developments • Increased attention to utilization of evaluation results • Focus on Evidence-Based Practice • Technology affects evaluation research • Topics to cover • Software to analyze data collected • Presentation of findings (e.g., word-processing and graphics capability) • New techniques for data gathering (e.g., Web surveys) • New skills and competencies needed Laura Saunders

11. Evaluation Applied What to study… Laura Saunders

12. Evaluation Can Be Applied To • Collections • Staff • Facilities • Services • Products • Information Technology Laura Saunders

13. Collections People find what they want/need Quality/value Size Use patterns Online Systems Complaint analysis Ideal page design Navigation/location of needed information Tracking eye movement Use patterns Reference Service Accuracy Complaint/compliment analysis Satisfaction Service quality (e.g., approachability) Technical Service Efficiency (costs and time) Time: staff time to complete task Time: time spent waiting between process activities Quality Accuracy (e.g., typographical errors in bibliographic records) Workflow analysis Some Topical Areas of Evaluation Laura Saunders

14. Formative Providing feedback during the development or offering of a program or service Summative After the program or service is over Evaluation Can Be Laura Saunders

15. Formative Evaluation • How can the program or service be improved or made more efficient or effective? • To what extent are the goals, objectives, and “performance measures” realized? Laura Saunders

16. Summative Evaluation • Is program or service X worth continuing or expanding? • How effective is it? • What conclusions can be drawn about its effects or various components? • Did it meet its goals, objectives, and performance expectations? Laura Saunders

17. Measuring Things… How much, How well, How fast, How many… Laura Saunders

18. Concept Measurements of Evaluation • Extensiveness, or the amount of a service provided in relation to the population served. This criterion is generally a measure of quantity rather than quality • Effectiveness • Efficiency, or the appropriateness of resource allocations • Cost-effectiveness • Cost-benefit • Quality Laura Saunders

19. Continuous Quality Improvement • General Systems Model • Input (staff, budget, resources) • Throughput (processes) • Output (services, collections) • Feedback to Input (evaluation) • All of above viewed within the context of the planning process Laura Saunders

20. Uses of Evaluation • Continue or discontinue (or improve) services (programs) • Improve, add, or drop practices and procedures (strategies and techniques) • Institute programs developed elsewhere • Allocate resources among competing services or programs • Accept or reject a program approach or theory Laura Saunders

21. Types of Evaluation Activities • Program Planning: focuses on • “What is the extent and distribution of the target population?” • “Is the program or service designed in conformity with its intended goals?” • “Are the chances of successful implementation maximized?” • Program (Service) Monitoring: focuses on • “Is the program or service reaching the persons, households, or other target units to which it is addressed?” • “Is the program or service providing the resources and other benefits that were intended in the project design?” Laura Saunders

22. Types of Evaluation Activities • Impact assessment: focuses on • “Is the program or service effective in achieving its intended goals?” • “Can the results of the program or service be explained by some alternative process that does not include the program?” • “Is the program or service having some effects that were unintended?” • Economic efficiency: focuses on • “What are the costs of delivering services and benefits to program participants?” • Does the program represent and efficient use of resources in comparison to alternative uses of the resources?” • Source: Rossi and Freeman, Systematic Evaluation Laura Saunders

23. Evaluation Questions All you need to do is ask. Laura Saunders

24. Evaluation Centers on “How” Questions Joseph Matthews Laura Saunders

25. Examples of Evaluation Questions • Are the services of the Libraries appropriate considering the needs of the university, its academic programs, the faculty, and students? • Are the services effectively delivered? • Are there important expectations or interests of the academic programs that the Libraries do not meet? Laura Saunders

26. Examples Cont’d • What are the specific strengths and weaknesses in libraries services at XXX? • Are there good lines of communication within the Libraries? Is staff morale good? • Are there adequate provisions for professional development of staff? • Is there adequate succession planning within a framework of future organizational structures? • Are there adequate and appropriate avenues for users to communicate needs and desired services back to the Libraries? Laura Saunders

27. Examples Cont’d • Is appropriate and sufficient attention being paid to anticipating future IT needs and trends? • Are there healthy and productive relationships with other academic service units on the campus? • Are staff adequately trained in and familiar with new technological directions? • What will be the most important qualities sought in the next director of libraries? Laura Saunders

28. If it’s so easy… Why isn’t everyone doing it? Laura Saunders

29. Barriers & Obstacles • The belief that everything is fine in my library • The library might apply evaluation inappropriately or merely serve organizational or administrative self-interests • Justify a weak or bad program or service by deliberately selecting for evaluation only those aspects that “look good” on the surface. Appearance replaces reality • Cover up program or service failure or errors by avoiding objective appraisal. Vindication replaces verification Laura Saunders

30. Barriers Cont’d • “Torpedo” or destroy a program or service regardless of its effectiveness. Politics replaces research • Treat evaluation as a “gesture” of objectivity or professionalism. Ritual replaces research • Delay needed action by pretending to seek the “facts.” Research replaces decision making • Perceived lack of resources to support the evaluation process • Etc. Laura Saunders

31. Barriers Cont’d • Lack of staff • Space • Budget • Conflicting perceptions of library • Limited staff skills • Physical location • Limited bibliographic control • Apathy of clientele and stakeholders about library • Poor organizational structure, planning, and decision making Laura Saunders

32. Preparing for Research Laura Saunders

33. Overview of the Evaluation Process • Phase One: Preparation • Establish library goals and objectives • Train staff • Understand evaluation • Put in place team or however the study will be done • Decide on the problem • Phase Two: Evaluation Research • Set up and execute the study • Phase Three: Assessing and, as decided, applying the results Laura Saunders