Effective Methods for Assessing the Impact of Service-Learning on Students, Institutions, and Communities

1. Effective Methods for Assessing the Impact of Service-Learning on Students, Institutions, and Communities ANDREW FURC0 University of Minnesota February 27, 2009

2. Problems with Service-Learning Evaluation • Program goals are unrealistic. • Expected outcomes cannot be easily measured. • Evaluation is an after thought. • Evaluation is not aligned to program goals • We claim impacts when we only measured outcomes. • The time frame is not commensurate for impacts to manifest. • We view all service-learning experiences to be the same . • We make bold claims after one study.

3. General Tips for Successful Evaluation • Set measurable, realistic goals • Focus the evaluation • Have your evaluation design match evaluation question(s) • Assess the quality of the service-learning practice • Secure an appropriate sample • Have a realistic timeline for data collection • Apply appropriate instruments & data collection techniques • Use a systematic approach for data collection • Have a plan for how the data will be analyzed systematically

4. Example DIFFICULT TO EVALUATE: To what extent did the service-learning experience reduce homelessness? BETTER: In what ways did the homeless situation change during the s-l experience?

5. Example DIFFICULT TO EVALUATE: What are the effects of service-learning on students’ development of civic capacities? BETTER: Which civic capacities do students report having gained from their engagement in service-learning?

6. Service-Learning Quality Matters • Principles (e.g., Wingspread Principles) • Balance Beam • Quadrant • Essential Elements

7. Service-Learning: A Balanced Approach to Experiential Education FOCUS Service Learning PRIMARY INTENDED BENEFICIARY Recipient Provider Service-Learning Community Service Field Education Volunteerism Internships

8. Assessing Service-Learning Quality High Service II I Unrelated Learning Integrated Learning III IV Adapted from the Service Learning Quadrant, Service Learning 2000 Center Low Service

9. Students in a Biology course are studying the brain and the nervous system. Students in the class are required to spend one afternoon during the month observing doctors at a neurology center. Students observe the kinds of treatments doctors offer patients with neurological disorders. At the end of the course, students write a brief paper in which theydiscuss their observations. High Service II I Unrelated Integrated Learning Learning III IV Low Service

10. Students in a Scandinavian Languages course spend three to five hours each week visiting with elderly individuals of Scandinavian descent. The students assist the seniors with daily tasks, such as reading the mail or newspaper, shopping, and exercising. The students also provide the seniors with an opportunity to share stories about the seniors’ life in Scandinavia. From this information, the students develop oral histories of Scandinavian life in the mid-Twentieth Century and preserve the histories in a monograph. High Service II I Unrelated Integrated Learning Learning III IV Low Service

11. ESSENTIAL ELEMENTS OF SERVICE-LEARNING CLUSTER 1: LEARNING 1) CLEAR EDUCATIONAL GOALS 2) CHALLENGING EDUCATIONAL TASKS: 3) ONGOING ACADEMIC ASSESSMENT CLUSTER 2: SERVICE 4) MEANINGFUL SERVICE TASKS 5) EVALUATION OF SERVICE OUTCOMES From Essential Elements of Service-Learning. National Youth Leadership Council. Minneapolis, MN.

12. Essential Elements of Service-Learning CLUSTER 3: CRITICAL COMPONENTS THAT SUPPORT SERVICE AND LEARNING 6) STUDENT VOICE 7) DIVERSITY OF PERSPECTIVES 8) OPEN AND EFFECTIVE COMMUNICATION 9) STUDENT PREPARATION 10) ONGOING AND CRITICAL REFLECTION 11) CELEBRATION .

13. Assessing Student Outcomes

14. Service-Learning and Academic Outcomes Mediating Factors Self-esteem Empowerment Prosocial behaviors Motivation Engagement Academic Outcomes Service- Learning Clearly defined programmatic features

15. MEASUREMENTS FOR STUDENT OUTCOMES • CART (http://cart.rmcdenver.com/)

16. Bringle, Phillips, and Hudson (2004): Contains a series of validated scales used in service-learning studies.

17. MEASUREMENTS FOR STUDENT OUTCOMES • CART (http://cart.rmcdenver.com/) • Bringle, Phillips, and Hudson (2004) • Higher Education S-L Survey (Civic Responsibility, Academic Attitude, Career Development, and Empowerment) (Furco, 2000) • CASQ (Civic Action, Social Justice, Appreciation of Diversity) (Moely et. al., 2002)

18. ISSUES IN S-L STUDENT ASSESSMENT • Student service preferences matter

19. Individual Preferences for Service

20. ISSUES IN S-L STUDENT ASSESSMENT • Student service preferences matter • Service-learning extends student learning

21. Aspects of Student Learning in Service-Learning Civic Responsibility Academic Achievement Community-based Learning Experiences LEARNING ABOUT SERVICE Connecting Learning and Service through Reflection LEARNING ABOUT THE SOCIAL ISSUE Classroom-based Learning Experiences LEARNING THE COURSE CONTENT …Service …Learning

22. Students learn the process of photosynthesis, the development of nutrients in plants, and the life cycles of various types of plants. . Learning Aspects of Service-Learning Plant Biology Edible Garden for the Homeless Learning the Course Content Students learn: to cultivate various edible plants; the nurturing that different types of plants require; assessment of plant quality; and garden care. Learning about Service Students learn: why and how individuals become homeless; the ways in which the homeless can benefit from being served the food from the gardent; the nature of homeless services in the local area, etc. Learning about the Social Issue

23. Assessing Institutional Outcomes • Institutionalization • Institutional climate (e.g. sense of community) • Learning communities • Faculty engagement in service-learning • Other

24. Instruments to Measure Institutionalization(Engagement and Service-Learning) • Kellogg Forum Checklist • Committee on Institutional Collaboration Benchmarks • Furco Rubric for Institutionalizing Service-Learning • Holland Matrix on Relevance to Mission • Campus Compact Indicators of Engagement 7) Bringle et al.’s Comprehensive Assessment for the Scholarship of Engagement (CASE) 8) Carnegie Elective Classification for Community Engagement

25. KELLOGG FORUM:DIMENSIONS OF ENGAGEMENT A CHECKLIST of ten principles: • Access to Learning • Enhanced Diversity • Civic Leadership • Public Scholarship • Social Well-Being • Trusted Voice • Public Spaces • Community Partnerships • Self Governance • Public Accountability

26. CIC BENCHMARKS OF ENGAGEMENT 7 BENCHMARKS that show evidence of: • Institutional commitment to engagement • Institutional resource commitments • Student Involvement in engagement activities • Faculty and staff partnerships with community • Institutional engagement with community • Assessing impact & outcomes • Resource/Revenue opportunities generated Committee on Institutional Cooperation

27. CIC BENCHMARKS OF ENGAGEMENT 1. Evidence of Institutional Commitment to Engagement 1.1. The institution’s commitment is reflected throughout its administrative structure. 1.2. The institution’s commitment is reflected in its reward structure for faculty and staff. 1.3. The institution’s commitment is reflected in its policies and procedures designed to facilitate outreach and engagement activities. 1.4. The institution’s commitment is reflected in its policies and procedures that are responsive to students. 2. Evidence of Institutional Resource Commitments to Engagement 2.1. The institution shows evidence of leadership for engagement and outreach activities. 2.2. The institution shows evidence of financial support for engagement through its budgetary process. 2.3. The institution shows evidence that faculty and staff time is devoted to outreach and engagement activities. 2.4. The institution includes engagement activities as part of its programs for faculty, student and staff development.

28. CAMPUS COMPACT INDICATORS OF ENGAGEMENT INDICATORS that suggest the presence of engagement at the institution: • Mission and Purpose • Academic and Administrative Leadership • Disciplines, Departments, and Interdisciplinary work • Teaching and Learning • Faculty Development • Faculty Roles and Rewards • Support Structures and Resources • Internal Budget & Resource Allocations • Community Voice External • Resource Allocation • Coordination of Community-Based Activities • Forums for Fostering Public Dialogue • Student Voice

29. OPERATIONALIZATION OF INDICATORS OF ENGAGEMENT A. Mission and Purpose • The institution’s mission statement explicitly articulates its commitment to the public purposes of higher education and is deliberate about educating students for lifelong participation in their communities. • This aspect of the mission is openly valued and is explicitly used to promote and to explain the civic engagement and community building activities on and off campus. • The institution demonstrates a genuine willingness to review, discuss, and strengthen its commitment to civic engagement and community building. • All members of the campus community demonstrate their familiarity with and ownership of the institution’s mission. From Burack and Saltmarsh (2007), Advancing Civic Engagement through Strategic Assessment. NERCHE

30. FURCO SELF-ASSESSMENT RUBRIC FOR SERVICE-LEARNING INSTITUTIONALIZATION A RUBRIC built on 22 components that are organized within five dimensions: • Philosophy and mission of service-learning • Faculty support and involvement in service-learning • Student support for and involvement in service-learning • Community participation and partnerships • Institutional support for service-learning Three stages: Critical mass building; Quality building; Sustained institutionalization

32. FINDINGS AND LESSONS LEARNED Based on data from 209 Colleges and Universities: • Overall Yr. 1 institutionalization level:  =1.6, s.d. = .21 Overall Yr. 3 institutionalization level:  = 2.1, s.d. = .31 Year 1-3 change:  = .50, p = 0.018 • Estimated 3-5 years to move from one stage to next • Internal assessments are more positive than external assessments 3 out of 4 cases (n = 43) in Year 1. Internal assessments are more negative than external assessments 3 out of 4 cases (n = 42) in Year 3.

33. FINDINGS AND LESSONS LEARNED • The self-assessment process is more important than the rubric score • The purposeful, ambiguous wording of the rubric encouraged discussion and dialogue 6) The components of the rubric are interdependent 7) All components do not have to be at Stage Three to achieve institutionalization 8) As you progress you may regress

34. ASSESSING COMMUNITY OUTCOMES

35. STRATEGIES FOR EVALUATING COMMUNITY OUTCOMES • Involve community partners/members in the design of the evaluation • Narrow the impact area to isolate service-learning’s effect • Secure pre-post measures; benchmark change • Support personal stories with data • Use pictures to support the data