Derek Rodriguez Doctoral Candidate, School of Information and Library Science

1. The ‘Understanding Library Impacts’ protocol:demonstrating academic library contributions tostudent learning outcomes in the age of accountability Derek Rodriguez Doctoral Candidate, School of Information and Library Science The University of North Carolina at Chapel Hill Program Officer, Triangle Research Libraries Network 9th Northumbria International Conference on Performance Measurement in Libraries and Information Services, August 23, 2011, York, England

2. Undergraduate education in the age of accountability Concerns Metrics Retention Graduation rates Student learning outcomes • Access and cost • Attainment • Competencies of graduates

3. Student learning outcomes • Locally defined learning outcomes • General education and the academic major • Broad abilities and cross-discipline skills • e.g. Critical thinking, inquiry and analysis, information literacy, quantitative literacy, written communication • VALUE rubrics (Rhodes, 2010) • Discipline specific knowledge and skills • By academic major, e.g. History, Chemistry, Nursing, etc. • Tuning projects (Gonzalez & Wagenaar, 2005; ICHE, 2010) - Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities. - González, J. and Wagenaar, R. (eds.) (2005). Tuning Educational Structures in Europe II. Bilbao, ES: University of Deusto. - Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

4. The challenge for libraries - Connecting library use to student learning outcomes Student Learning Student effort Library use Assessment ? Program reviews External frameworks VALUE & Tuning

5. ‘Understanding Library Impacts’- Explores library use as a component of student effort Student Learning Learning Activities Library use Assessment Student effort Program reviews External frameworks VALUE & Tuning

6. A focus for library assessment Capstone experience Expectations Academic major Student effort General education

7. Design for pilot study (spring 2011) • Population: undergraduate history majors enrolled at 2 sites, a liberal arts college and a liberal arts university in the U.S. (N=74) • Project: 20 – 40 page research paper using evidence from primary sources • Step 1: Constructed a learning activities crosswalk using syllabi and rubrics for research papers • Step 2: Students completed a web-based critical incident survey after turning in papers • Step 3: Reports were distributed to study sites via secure, database-driven web-sites

8. Understanding Library Impacts- Explores library use as a component of student effort Student Learning Learning Activities Library use Assessment ‘Capstone’ project Program reviews Learning activities crosswalk Critical incident survey External frameworks VALUE & Tuning

9. Learning activities crosswalk Activities Capstone-related outcomes Getting oriented • Discipline-specific skills • Locates secondary and primary sources • Distinguishes among types of sources Choosing a topic • Thesis and argument • Develops original thesis statement • Advances argument in support of thesis using evidence from primary sources Developing a thesis Gathering evidence • Evidence and analysis • Evaluates and interpret primary sources • Uses secondary sources to provide context • Demonstrates an understanding of the methods of history Finding other sources Creating a bibliography • Writing and Citing • Communicates argument in a coherent, well-written paper • Follows disciplinary style and citation standards Writing

10. Critical incident survey- Developed and refined in qualitative studies (2006, 2007) Deliverables and outcomes Learning activities Helps and problems Open ended questions and probes Library use Top-ranked e-resource - Electronic resources Local questions Top-ranked traditional resource - Traditional resources Demographics and affect Top-ranked service - Services Academic challenge Top-ranked facility - Facilities / Equipment

13. Outcomes related to the activity ‘Gathering evidence’ • Locates secondary and primary sources • Distinguishes among types of sources • Advances argument in support of thesis using evidence from primary sources • Evaluates and interprets primary sources

14. Outcomes related to the activity ‘Writing’ • Communicates argument in a coherent, well-written paper • Follows disciplinary style and citation standards • Advances argument in support of thesis using evidence from primary sources • Uses secondary sources to provide context

15. Factors of use (helps and problems) • Derived from qualitative studies and LIS literature • Gathered for ‘ranked uses’ only • Categorized for analysis • Example category: ‘Help finding information’ • “I learned about information sources for my project” (help) • “I learned new skills” (help) • “helped me when I got stuck” (help) • “the assistance I received wasn't helpful” (problem) • “it was difficult to find someone to help me” (problem)

17. Challenges faced during the project Site B, n=22 Site A, n=17 “Overwhelmed by various themes and layers of my research” (A-2) “Putting all of my information together into a cohesive paper” (A-5) “Time management” (A-8) “Several of the books I needed were being rebound…” (B-3) “Uncertainty about where to start” (B-10) “I was worried about finding primary source material” (B-15) “At a somewhat early point in the project, I realized that the scope of my project was much too wide …” (B-14)

18. Most important library uses, by type of use, n=37 “… those databases, JSTOR specifically, really helped me.” (B-4) “Library space to work on my thesis both writing and research” (B-2) “being able to get books from off campus” (B-3) Alternate activity, if first choice hadn’t been available, n=29

19. Understanding Library Impacts protocol- Contributions and implications Student Learning Library use Assessment Learning Activities ‘Capstone’ project Program reviews - Credible connections between library use and expectations for student learning - Generates rich data useful for advocacy and improvement - Places library in important campus conversations about student learning External frameworks VALUE & Tuning

20. Next steps Refining instruments; conducting reliability and validity testing Exploring “patterns and predictions” Continue evaluation with history majors at 4 more sites (fall 2011) Future work Evaluate protocol with other disciplines and settings Explore integration with assessment systems and analytics projects Questions? Contact: Derek_Rodriguez@unc.edu

21. Crosswalk: Broad abilities and discipline-specific outcomes • VALUE Rubrics (Rhodes, 2010) • Critical thinking • Inquiry and analysis • Information literacy • Written Communication • Tuning Outcomes for History (ICHE, 2010) • Historical knowledge • Thinking and analytical skills • Communication skills • Personal motivation Rhodes, T., ed. 2010. Assessing Outcomes and Improving Achievement: Tips and Tools for Using Rubrics. Washington, DC: Association of American Colleges and Universities. Indiana Commission for Higher Education. Tuning USA Final Report: The 2009 Indiana Pilot, 2010.

22. Related capstone outcomes • Locates secondary and primary sources • Distinguishes among types of sources • Advances argument in support of thesis using evidence from primary sources • Evaluates and interprets primary sources

23. Outcomes related to ‘gathering evidence’ Project outcome Advances argument in support of thesis using evidence from primary sources VALUE Inquiry and Analysis Analysis - Organizes and synthesizes evidence to reveal insightful patterns, differences, or similarities. Activity Gathering evidence Tuning Outcome Formulate and test plausible historical hypotheses and marshal an argument.