Assessing Co-curricular Learning

1. Assessing Co-curricularLearning Robert Mundhenk Visiting Scholar The Higher Learning Commission

2. Rescheduled for Jan. 19 at 3:30 – 5:00 p.m. YU 204 • Co-curricular Assessment workshop presented by Bob Mundhenk                 Yellowjacket Union 203  While most institutions have devoted effort and resources to the effectiveness of the co-curricular services and programs they offer, only recently have they started to assess the learning these areas produce. This session will explore ways to assess the impact of co-curricular efforts on student learning and development. Targeted Audience:  Faculty and staff in non-academic units interested in assessment.      • Ask Institutional Research about graduation, retention, GPAs, and the like • Ask faculty about their teaching and the learning it produces—but not necessarily whether they know they’re producing learning • Ask faculty how they know they’re producing learning Traditional Approaches to Assessment of Learning

3. Traditional wisdom: Learning outcomes need to be aligned at course, program, and institutional levels • But where are many general education goals, like “tolerance” and “teamwork” and “the ability to function in an increasingly diverse world” and “inclination” taught and assessed? • Or is “taught” the right word??? The Institutional Mismatch

4. What happens if we substitute the word “learned” for the word “taught”? • What are the implications of “Where are learning outcomes learned and assessed”? • Emphasis on student demonstration, not topic-covering • Ability to do or apply supersedes knowing • Responsibility for learning is shared • Site of learning becomes less specific, and boundaries become more fungible Shifting Perspectives

5. “Learning” is not exclusively classroom-based • Many valued outcomes are not taught exclusively in the classroom • Many valued outcomes are the result of processes outside the classroom • “Learning” is a process based on three interdependent student experiences: • Understanding academic content and processes • Student development • Identity formation After Learning Reconsidered

6. Responsibility for “learning” exists outside the classroom • Responsibility for “learning” doesn’t always take the same form; some entities on campus produce it, some facilitate it, some support it • Responsibility for assessing learning exists outside the classroom as well After Learning Reconsidered

7. Civic Responsibility • AA: Service learning • SA: Student government, voter registration, student judicial boards • Think and Engage as a Global Citizen • AA: Language courses, Anthropology, Sociology • SA: International experiences, culture days, residence halls Some Post-LR Examples of Learning

8. Ability and inclination to: • Think and make connections across disciplines • Express oneself in multiple forms • Analyze and reflect upon multiple perspectives to arrive at a perspective of one’s own • Think and engage as a global citizen • Engage in evidence-based problem-solving UW- Superior’s Five Institutional Goals

9. How is ability made evident? • Where and how is ability developed? • Where is ability assessed? • What is “inclination”? • How is it made evident? • Where and how is it developed? • Where is it assessed? Ability and Inclination

10. Campus Life learning outcomes: • Recognizing and enhancing leadership potential • Developing an appreciation of human differences • Seeking opportunities to engage in the campus community and beyond • Expanding desire for life-long learning How Do These Outcomes Connect with Undergraduate Learning Outcomes?

11. Campus Life learning outcomes: • Recognizing and enhancing leadership potential • Developing an appreciation of human differences • Seeking opportunities to engage in the campus community and beyond • Expanding desire for life-long learning What Ability or Inclination Do They Develop?

12. Campus Life learning outcomes: • Recognizing and enhancing leadership potential • Developing an appreciation of human differences • Seeking opportunities to engage in the campus community and beyond • Expanding desire for life-long learning How Do We Know They’re Achieved?

13. Need to be intentional • Need to be planned • Need to be part of the structure of a student’s experience • Need to be assessed Co-curricular Outcomes

14. BEING INTENTIONAL

15. Efficiency models: • Focus on process • How well is this office/service functioning? • Focus on numbers: • Clients served • Graduation rates • Tutorial visits • Attendance at activities • Student/staff ratios Traditional Co-curricular Assessment

16. Effectiveness Models: Indirect • Based on surveys and other indirect indicators, like NSSE • Often rely on student self-reporting • Tend to skew positively on outcomes, if not always on the processes that led to them Newer Co-curricular Assessment

17. Effectiveness Models: Direct • Focuses on student performance • Can be based on observation or objective measures • Require carefully designed and consistent measuring practices Newer Co-curricular Assessment

18. Apply external standards, like CAS • Use surveys and questionnaires • Develop direct measurement strategies • All of the above How to Assess Co-curricular Learning

19. Standards for 40 functional areas • Thirteen component parts: • Mission • Program • Leadership • Organization and management • Human resources • Financial resources • Facilities, technology, and equipment • Legal responsibilities • Equity and access • Campus and community relations • Diversity • Ethics • Assessment and evaluation CAS Standards

20. Knowledge acquisition, construction, integration, and application • Cognitive complexity • Intrapersonal Development • Interpersonal competence • Humanitarianism/Civic Engagement • Practical Competence CAS’s Six Outcome Domains

21. Can easily document the efficiency of processes and organization • Can be used as well (through an emphasis on the domains) to chart the effectiveness of outcome development efforts—depending on outcome definition and quality of evidence Using CAS Standards

22. Career Services: As a result of interactions with the Career Services Office, students and alumni will: • Identify their skills, abilities, and strengths in order to make knowledgeable career decisions • Have the necessary resources and skills to prepare for life-long post-graduate experiences • How are these outcomes connected to institutional learning outcomes? • What are “resources and skills”? Outcome Definition

23. College Unions: As a result of experiences in the Yellowjacket Union, students will: • Identify and utilize the opportunities and services available to them • Demonstrate a sense of ownership for the campus community and civic involvement • Interact with and value individuals from diverse backgrounds and lifestyles • How do you know you’ve achieved the second and third outcomes? • How do they overlap with institutional outcomes? Outcome Definition

24. Clearly defined outcomes help determine the nature of evidence to be collected • Clearly defined outcomes focus on student performance and development, thus calling for both direct and indirect forms of evidence Outcome Definition and Evidence

25. DIRECT: uses performance or product, created by students, that can be compared to expected outcomes --Capstone courses, projects, internships, clinical experiences, etc. INDIRECT: uses information that does not directly link the learning to the outcomes --graduation rates, grades, surveys, “usage” rates Direct and Indirect Measures

26. Is it relevant to the area’s stated mission and function? • Does it measure what we want it to measure? • Does it deal in some way with outcomes? • Is the information derived useful? • Can the information be used to improve either function or learning? Questions about Evidence

27. Did you accomplish what you hoped to accomplish in your meeting with your advisor? • How well did your experience at X prepare you for employment? • As a result of this First-Year program, do you feel better prepared for college? • Write a short essay in which you describe the ways in which your attitudes and values have changed as a result of your semester in Argentina. Surveys and Questionnaires: Some Sample Questions

28. Traditional, indirect source of information on effectiveness • Limitations: • Self-reporting • Unvalidated opinion • Response rates • Opportunistic data • Skewed samples Surveys and Questionnaires: Indirect Evidence

29. Kinds: • Satisfaction • Reflective • Post-experience experience (alumni and employers) • Value: • True “customer” response • Can indicate areas for improvement and ratification • Provides data for planning and review Surveys and Questionnaires

30. Making them tools to assess learning: • Use learning outcomes as basis for at least some questions • Validate by cross-referencing outcomes with different populations (employers, alumni, graduate, current students) • Emphasize the learning outcomes in design and analysis of surveys and questionnaires Surveys and Questionnaires

31. If Learning Reconsidered made the case for cross-campus responsibility for learning, then assessment of learning outcomes is also a cross-campus responsibility • Adaptation of practices and devices already in use in academic settings • Standardized • Judgment-based Direct Effectiveness Measures

32. Intentional Planning: • Determine areas of responsibility: what office/function might be a logical place to contribute to particular learning outcomes? • Plan the outcome-based purpose of the activity • Aim at the appropriate level of Bloom’s taxonomy • Design non-passive activities (watching a film plus discussion; International Days as more than food, costumes, and dance) • Design outcome-focused opportunities for processing Developing Direct Measures of Effectiveness

33. What is “inclination” and how is it developed in co-curricular activities? • How does a student “express oneself in multiple forms”? • What is a “global citizen”? • What does it mean to “make connections across academic disciplines” and how might co-curricular activities have a role in developing this outcome? Understanding the Outcomes

34. Choose one of the outcomes below and determine how your co-curricular area might have some responsibility for developing it. Name specific activities that might help develop the outcome and specify what their effect on the student should be. • Think and make connections across academic disciplines • Express oneself in multiple forms • Analyze and reflect upon multiple perspectives to arrive at a perspective of one’s own • Think and engage as a global citizen • Engage in evidence-based problem-solving Being Intentional

35. Mapping: • If the learning outcome is important, single exposure isn’t enough • How do first-year experiences differ from last-year ones—or what difference is expected in student response? • How to assure student’s development of outcomes from first year to last? Planning for the Long Term

36. One-shot assessment produces haphazard results that are usually insufficient for planning improvement • Tie assessments to logical stages of development, based on an outcome map • Be consistent in approach to assessing • Options: • Standardized instruments • Self-generated tools Regular Assessment

37. Observations • Expert judgments • Student self-reflection • Employer/supervisor judgments Self-generated Tools

38. Consistency across observers is crucial, so a rubric of some kind is essential • Holistic rubrics: broad judgments (Acceptable/Not Acceptable/Needs Improvement) • Descriptive rubrics: defined criteria and measures Using Self-Generated Tools

39. First, determine the aspects of student performance that would indicate he/she has achieved an outcome (e.g., one aspect of a “social justice” outcome might be “the student’s writing demonstrates sensitivity to issues of class and power”) • Second, define the specific things a student would have to do to show he/she has mastered that aspect (e.g., “Clear understanding of the ways in which economic status affects behavior.” (Criteria) • Finally, describe degrees of achievement for each criterion (Measures) Descriptive Rubrics

40. Using the outcome and the functional area you chose earlier, develop a rubric to measure student achievement, defining one performative aspect of the outcome, one criterion for measuring that aspect, and a system (holistic, descriptive, whatever) for describing degrees of attainment Developing a Rubric

41. Assessments done across campus should ideally use the same rubrics or measures • When using the same tools is not possible, it is essential that there be a way to extract information that is usable at the institutional level while still serving the needs of the functional area Institutional Assessment

42. To conclude: • Understand the meaning of the desired outcomes and your role in developing them • Separate efficiency from effectiveness • Plan experiences and assessments carefully • Focus on using assessment information to improve learning, not to justify your existence • Collect information that is relevant, meaningful, and useful • Design systems that are reasonable and manageable

43. ROBERT MUNDHENK asmt357@aol.com rmundhenk@hlcommission.org