Assessment in service learning: Student learning outcomes

1. Assessment in service learning: Student learning outcomes Julie A. Hatcher, Exec. Dir. Assoc. Prof. Philanthropic Studies jhatcher@iupui.edu Kristin Norris, ABD norriske@iupui.edu Assessment Director Center for Civic & Social Responsibility University of Kansas May 21, 2014

2. Outcomes for work/play shop • Basic intro to assessment (of x, y, z) • Implications of a “public work” approach at KU • IUPUI “case study” • CMG • NSSE • Identify 1-2 AAC&U learning outcomes for either • Course • Program • Campus • Identify next steps you will take on your assessment matrix

3. Current Support for “Civic” • Membership Organizations • AAC&U • AASC&U • American Academy of Arts and Sciences • Campus Compact • Imagining America • NASPA • Foundations • Bonner Foundation • Carnegie Foundation: Elective Classification for C.E • Kettering Foundation • Lang Foundation • Lilly Endowment • Lumina Foundation • Teagle Foundation • John Templeton Foundation • Government • State Commissions of Higher Education • Massachusetts Board of Higher Education

4. Rigor in Research (Assessment) • Rigorous thinking • Hypothesis exploration • Systematic approach • Builds upon prior research • Contributes to the field • Triangulation across inquiry • Information synthesis • Creates useful, relevant knowledge (Patton, 2012)

5. Schoen’s Reflective Practice • Trait of a good professional • Reflection-in-action • Attentive to new knowledge • Test, adjust, reframe their models of practice • Revise plans

6. Kolb’s Experiential Learning Cycle Concrete Experience Active Experimentation Reflective Observation Abstract Conceptualization

7. Reflective SL Practitioner Practice SL Course/Program/ Curriculum Design Formative Assessment Reflection Activities Classroom Assessment Implications for you/others Program/Course Redesign Theory Research/SoTL

8. My Reflective Observations • Wine or peanut M&Ms • Important to identify a “north star” • Takes a significant amount of time • Year of Assessment in 1997 ish • Highly connected to scholarship • We find our way by walking… • 3 semesters until “in the saddle” • Refine, improve reflection strategies • Ongoing improvement of processes • Tweaks

9. Boyte’s concept of Public Work • Places public work (not deliberation or social justice) at the center of democracy • Common pool resources (Ostrom School) • Collective action – “we” not “they” • Lay participation is essential • Trait of a democratic professional (Dzur) • Conflict is intrinsic and healthy aspect • Co-generators of knowledge -- partners

10. Assessment as Public Work • Portland State case study -- resources • Assessing service-learning and civic engagement: Principles and techniques • AAC&U – VALUE rubric case study • 16 rubrics; all available free • No need to reinvent the wheel • IUPUI case study

11. Current Context at KU • Reflect – pair & share – wagon wheel • What are the key “pressures” or levers that are motivating you personally to be here today? • What do you hope to do, improve, advance as a result of being here today? • Who do you need to bring to the table so this is done collectively?

12. Terms of Scholarship -- data • Assessment (What occurred?) • Formative – during the course/program • Summative – end of the course/program • Program Evaluation (What aspect?) • Program Evaluation Research • Research (Why? What conditions?) • Qualitative • Quantitative • Mixed-methods

13. Definition Service learning is a course-based, credit-bearing educational experience* in which students a) participate in an organized service activity that meets identified community needs*, and b) reflect on the service activity in such a way as to gain • further understanding of course content, • a broader appreciation of the discipline, and • an enhanced sense of personal values* and civic responsibility. • (Bringle & Hatcher, 1995)

14. Formative Assessment in SL • Classroom Assessment Techniques (Angelo & Cross, 1995) • Exit Cards • Minute Papers • Mid-semester evaluation • Reflection activities • Pair and Share • Journals - various types (DEAL model) • Value of informal/unstructured • Community partner feedback • Coffee……and conversation

15. Summative Assessment -- Data • Course-based • Grades • End-of-Course survey; pre/post • Products • Curriculum-based • E-portfolio • Digital stories • Exit survey for majors • University-based • Counting; classifying • Longitudinal impact on students • Evidence of community impact

16. Assessment in SL Course • Are grades sufficient? (perhaps) • What literature review could help to “ground” the assessment? (ex. Nutrition, Architecture) • How can you use/analyze reflection activities as a source of data to document student learning? • What pre/post learning could you assess? • What other products could you gather? • Multiple sources of data -- triangulation

17. Designing and Assessing SLClass/Program Learning Objectives of Course - Curriculum - Campus Assessment of StudentLearning and Products Scholarship of Teaching/Learning SOTL Selection of Community Partner and Service Experience Structured Reflection Activities Framed by LearningObjectives Products are Created

18. Service Learning Outcomes • Academic Development • Persistence and retention • Achievement and aspirations • Life Skills • Racial tolerance • Cultural understanding • Civic Responsibility • Commitment to community • Aspirations to volunteer (Sax & Astin, 1997; UCLA/HERI) (www.compact.org/resource/aag.pdf)

19. Meta-Analyses of SL • Requires a body of research • Psychology, business, communications • Academic • Personal • Social • Citizenship

20. Case Study from IUPUI • Institutional culture and commitment • Assessment & Civic Engagement • Assessment Institute (Oct 21-23., 2014) • Ethos in our work • Bob Bringle’s leadership • Scholarship and research • Signature Center designation • IUPUI Book Series – Research on SL Vol 1, Vol 2 • Boyer Scholars Faculty Learning Community • IUPUI Research Academy (each May)

21. Types of CitizenshipAcross Disciplines • Civic professionalism • Social responsibility • Social justice • Connected knowing: Ethic of care • Public leadership • Public intellectual • Engaged/public scholarship (Battistoni, 2002)

22. Civic-Minded Graduates • “to what end? ” • Civic-Minded Professionals • John Dewey • Democracy and Education • William Sullivan • Habits of the Heart • Work and Integrity • Ernest Boyer • “public good” argument

23. CMG as a “North Star” • A civic-minded graduate is one who • is formally educated and • has the capacity and orientation to work with others • in a democratic way • to improve the community. • (Hatcher, 2011)

24. Civic-Mindedness “a person’s inclination or disposition to be knowledgeable of and involved in the community, and to have a commitment to act upon a sense of responsibility as a member of that community” (Steinberg, Hatcher, Bringle, 2011)

25. Civic-Minded Graduate Model

26. Civic and Workforce Development • Developing civic-minded graduates and professionals • Developing workforce “soft” skills • Recognizing the importance of place (Battistoni & Longo, 2012)

27. Domains of CMG: Civic Knowledge • More than purely academic knowledge(dates, places, important civic or political events) • Knowledge of volunteer opportunities(ways to contribute to society and of nonprofit organizations) • Knowledge of contemporary social issues (current events and the complexity of issues in modern society) (Steinberg, Bringle, & Hatcher 2011)

28. Domains of CMG: Civic Skills • Communication and Listening(ability to communicate with others and listen to divergent points of view) • Diversity (understanding the importance of, and the ability to work with others from diverse backgrounds) • Consensus-building(ability to work across difference to come to an agreement or solve a problem)

29. Domains of CMG: Civic Dispositions • Valuing community engagement (understanding the importance of service to others, and being actively involved in the community) • Self-efficacy (have the desire to take personal action, with a realistic view that the action will produce the desired result) • Social trustee of knowledge (feeling a sense of responsibility and commitment to use the knowledge gained in college to serve others)

30. CMG Survey • 30 Likert-type items (student self-report) • Knowledge, skills, dispositions, behaviors • 6-point response format • Demographic items • Student activity items • Paper or online administration • Adaptable: “My education at IUPUI” – course, or major, depends on the research question

31. Assessment Tools • Course Evaluation • SL End –of-Course Survey • Domains to include for your campus? • Student Learning Outcomes • Civic-Minded Graduate Scale • Civic-Minded Graduate Narrative • Civic-Minded Graduate Rubric • AAC&U VALUE Rubric (Civic Engagement)

32. Implications of CMG--CMP • Program Design • Alternative Break; days of service • Service-scholarship Applications • Plater Medallion • Consultations with faculty and departments • New work in faculty development and graduate education

33. Pair and Share • Put on your “disciplinary hat” • Why is it important to you as a…landscapearchitect, student affairs professional…to include civic learning in your course/program? • What type of civic learning do you hope occurs for your students? • Who are your stakeholders, and why would they care about these outcomes?

34. Learning Objectives • List 2 civic learning objectives in your community engaged program/course. • (How do these align with broader curriculum?) • (How do these align with accountability for external program review, if applicable?) • (How do these align with the campus mission for student learning?)

35. “Deep Learning” • High Impact Practices (e.g., “HIPs”) • Distinct value of service learning • Additive value of multiple experiences • See AAC&U, Ashley Finley, Director of Research and Assessment • NSSE Data (12 items) • Higher-Order learning • Integrative learning • Reflective learning

36. Participation in Service Learning Courses The independent variable, participation in serving learning courses, was derived from NSSE survey question 1K: In your experience at your institution during the current school year, about how often have you done each of the following? k. Participated in a community-based project (e.g., service learning) as part of a regular course

37. Participation in Service Learning Courses

38. Deep Learning • The dependent variable deep learning was comprised of three different scales. Reliability analysis was conducted for higher-order learning (α=.83), integrative learning (α=.73), and reflective learning (α=.83). • The data file was then split into freshman and senior students so the analysis could be conducted on these two populations separately.

39. Higher Order Learning Questions (α=.83)

40. Integrative Learning Questions (α=.73)

41. Reflective Learning Questions (α=.83)

42. The Findings • An independent-samples t-test evaluated differences in reported deep learning skills between students who participated in one or more service learning courses and those students who did not participate in service learning courses. • Deep learning skills of higher-order learning, integrative learning, and reflective learning were all higher for both seniors and freshman who participated in service learning course(s).

43. IUPUI Freshman *p<.05, 2-tailed significance

44. IUPUI Seniors *p<.05, 2-tailed significance

45. Implications • Results contribute evidence of student learning at the institution level • Findings are consistent with prior research on participation in service learning and improved student outcome measures (Astin et. al., 2000) • Provide a rationale for institutions to support faculty who engage with the community partners to develop service learning courses

46. Matrix Framework