Effective Methods for Assessing the Impact of Service-Learning on Students, Institutions, and Communities

1. Effective Methods for Assessing the Impact of Service-Learning on Students, Institutions, and Communities ANDREW FURC0 University of Minnesota February 27, 2009

2. Problems with Service-Learning Evaluation • Program goals are unrealistic. • Expected outcomes cannot be easily measured. • Evaluation is an after thought. • Evaluation is not aligned to program goals • We claim impacts when we only measured outcomes. • The time frame is not commensurate for impacts to manifest. • We view all service-learning experiences to be the same . • We make bold claims after one study.

3. General Tips for Successful Evaluation • Set measurable, realistic goals • Focus the evaluation • Have your evaluation design match evaluation question(s) • Assess the quality of the service-learning practice • Secure an appropriate sample • Have a realistic timeline for data collection • Apply appropriate instruments & data collection techniques • Use a systematic approach for data collection • Have a plan for how the data will be analyzed systematically

4. Service-Learning Quality Matters • Principles (e.g., Wingspread Principles) • Balance Beam • Quadrant • Essential Elements

5. Service-Learning: A Balanced Approach to Experiential Education FOCUS Service Learning PRIMARY INTENDED BENEFICIARY Recipient Provider Service-Learning Community Service Field Education Volunteerism Internships

6. Assessing Service-Learning Quality High Service II I Unrelated Learning Integrated Learning III IV Adapted from the Service Learning Quadrant, Service Learning 2000 Center Low Service

7. ESSENTIAL ELEMENTS OF SERVICE-LEARNING CLUSTER 1: LEARNING 1) CLEAR EDUCATIONAL GOALS 2) CHALLENGING EDUCATIONAL TASKS: 3) ONGOING ACADEMIC ASSESSMENT CLUSTER 2: SERVICE 4) MEANINGFUL SERVICE TASKS 5) EVALUATION OF SERVICE OUTCOMES From Essential Elements of Service-Learning. National Youth Leadership Council. Minneapolis, MN.

8. Essential Elements of Service-Learning CLUSTER 3: CRITICAL COMPONENTS THAT SUPPORT SERVICE AND LEARNING 6) STUDENT VOICE 7) DIVERSITY OF PERSPECTIVES 8) OPEN AND EFFECTIVE COMMUNICATION 9) STUDENT PREPARATION 10) ONGOING AND CRITICAL REFLECTION 11) CELEBRATION .

9. Assessing Student Outcomes

10. MEASUREMENTS FOR STUDENT OUTCOMES • CART (http://cart.rmcdenver.com/)

11. Bringle, Phillips, and Hudson (2004): Contains a series of validated scales used in service-learning studies.

12. MEASUREMENTS FOR STUDENT OUTCOMES • CART (http://cart.rmcdenver.com/) • Bringle, Phillips, and Hudson (2004) • Higher Education S-L Survey (Civic Responsibility, Academic Attitude, Career Development, and Empowerment) (Furco, 2000) • CASQ (Civic Action, Social Justice, Appreciation of Diversity) (Moely et. al., 2002)

13. ISSUES IN S-L STUDENT ASSESSMENT • Student service preferences matter

14. Individual Preferences for Service

15. ISSUES IN S-L STUDENT ASSESSMENT • Student service preferences matter • Service-learning extends student learning

16. Aspects of Student Learning in Service-Learning Civic Responsibility Academic Achievement Community-based Learning Experiences LEARNING ABOUT SERVICE Connecting Learning and Service through Reflection LEARNING ABOUT THE SOCIAL ISSUE Classroom-based Learning Experiences LEARNING THE COURSE CONTENT …Service …Learning

17. Students learn the process of photosynthesis, the development of nutrients in plants, and the life cycles of various types of plants. . Learning Aspects of Service-Learning Plant Biology Edible Garden for the Homeless Learning the Course Content Students learn: to cultivate various edible plants; the nurturing that different types of plants require; assessment of plant quality; and garden care. Learning about Service Students learn: why and how individuals become homeless; the ways in which the homeless can benefit from being served the food from the gardent; the nature of homeless services in the local area, etc. Learning about the Social Issue

18. Assessing Institutional Outcomes • Institutionalization • Institutional climate (e.g. sense of community) • Learning communities • Faculty engagement in service-learning • Other

19. Instruments to Measure Institutionalization(Engagement and Service-Learning) • Kellogg Forum Checklist • Committee on Institutional Collaboration Benchmarks • Furco Rubric for Institutionalizing Service-Learning • Holland Matrix on Relevance to Mission • Campus Compact Indicators of Engagement 7) Bringle et al.’s Comprehensive Assessment for the Scholarship of Engagement (CASE) 8) Carnegie Elective Classification for Community Engagement

20. KELLOGG FORUM:DIMENSIONS OF ENGAGEMENT A CHECKLIST of ten principles: • Access to Learning • Enhanced Diversity • Civic Leadership • Public Scholarship • Social Well-Being • Trusted Voice • Public Spaces • Community Partnerships • Self Governance • Public Accountability

21. CIC BENCHMARKS OF ENGAGEMENT 7 BENCHMARKS that show evidence of: • Institutional commitment to engagement • Institutional resource commitments • Student Involvement in engagement activities • Faculty and staff partnerships with community • Institutional engagement with community • Assessing impact & outcomes • Resource/Revenue opportunities generated Committee on Institutional Cooperation

22. CIC BENCHMARKS OF ENGAGEMENT 1. Evidence of Institutional Commitment to Engagement 1.1. The institution’s commitment is reflected throughout its administrative structure. 1.2. The institution’s commitment is reflected in its reward structure for faculty and staff. 1.3. The institution’s commitment is reflected in its policies and procedures designed to facilitate outreach and engagement activities. 1.4. The institution’s commitment is reflected in its policies and procedures that are responsive to students. 2. Evidence of Institutional Resource Commitments to Engagement 2.1. The institution shows evidence of leadership for engagement and outreach activities. 2.2. The institution shows evidence of financial support for engagement through its budgetary process. 2.3. The institution shows evidence that faculty and staff time is devoted to outreach and engagement activities. 2.4. The institution includes engagement activities as part of its programs for faculty, student and staff development.

23. CAMPUS COMPACT INDICATORS OF ENGAGEMENT INDICATORS that suggest the presence of engagement at the institution: • Mission and Purpose • Academic and Administrative Leadership • Disciplines, Departments, and Interdisciplinary work • Teaching and Learning • Faculty Development • Faculty Roles and Rewards • Support Structures and Resources • Internal Budget & Resource Allocations • Community Voice External • Resource Allocation • Coordination of Community-Based Activities • Forums for Fostering Public Dialogue • Student Voice

24. OPERATIONALIZATION OF INDICATORS OF ENGAGEMENT A. Mission and Purpose • The institution’s mission statement explicitly articulates its commitment to the public purposes of higher education and is deliberate about educating students for lifelong participation in their communities. • This aspect of the mission is openly valued and is explicitly used to promote and to explain the civic engagement and community building activities on and off campus. • The institution demonstrates a genuine willingness to review, discuss, and strengthen its commitment to civic engagement and community building. • All members of the campus community demonstrate their familiarity with and ownership of the institution’s mission. From Burack and Saltmarsh (2007), Advancing Civic Engagement through Strategic Assessment. NERCHE

25. FURCO SELF-ASSESSMENT RUBRIC FOR SERVICE-LEARNING INSTITUTIONALIZATION A RUBRIC built on 22 components that are organized within five dimensions: • Philosophy and mission of service-learning • Faculty support and involvement in service-learning • Student support for and involvement in service-learning • Community participation and partnerships • Institutional support for service-learning Three stages: Critical mass building; Quality building; Sustained institutionalization

27. FINDINGS AND LESSONS LEARNED Based on data from 209 Colleges and Universities: • Overall Yr. 1 institutionalization level:  =1.6, s.d. = .21 Overall Yr. 3 institutionalization level:  = 2.1, s.d. = .31 Year 1-3 change:  = .50, p = 0.018 • Estimated 3-5 years to move from one stage to next • Internal assessments are more positive than external assessments 3 out of 4 cases (n = 43) in Year 1. Internal assessments are more negative than external assessments 3 out of 4 cases (n = 42) in Year 3.

28. FINDINGS AND LESSONS LEARNED • The self-assessment process is more important than the rubric score • The purposeful, ambiguous wording of the rubric encouraged discussion and dialogue 6) The components of the rubric are interdependent 7) All components do not have to be at Stage Three to achieve institutionalization 8) As you progress you may regress

29. ASSESSING COMMUNITY OUTCOMES

30. STRATEGIES FOR EVALUATING COMMUNITY OUTCOMES • Involve community partners/members in the design of the evaluation • Narrow the impact area to isolate service-learning’s effect • Secure pre-post measures; benchmark change • Support personal stories with data • Use pictures to support the data