The Role of Assessment in Student Affairs Practice

1. Building a culture of evidence: The Role of Assessment in Student Affairs Practice CSA 502: Organization and Administration in Student Affairs February 10, 2011

2. Presentation outline • What is assessment & why do we do it? • The role of SARA and assessment offices in general • Assessment planning & the assessment cycle • It’s your turn - practice makes perfect!

3. Learning objectives: At the end of this session, you should be able to: • Identify why assessment is important in all aspects of student affairs. • Describe the general role of an assessment office. • Describe the assessment cycle. • Perform the basic steps to create an assessment plan.

4. What is assessment? • It is “any effort to gather, analyze, and interpret evidence which describes…effectiveness (Upcraft & Schuh, 996) • Upcraft & Schuh (2002) argue that research informs theory while assessment informs practice. • But what does that really mean in practical terms??

5. Why is assessment important? • Helps determine if you are meeting your educational objectives • Helps ensure that you have the resources you need • Helps prioritize efforts • Can contribute to our understanding of student learning and development

6. Student Affairs Research & Assessment (SARA) • Mission Student Affairs Research and Assessment provides leadership and service to the Division of Student Affairs in the areas of assessment, learning outcomes, educational programming, and strategic planning. SARA collects and disseminates data about Penn State students, their experiences and learning, and their perceptions of the campus environment. SARA enhances students’ educational experiences through the design, implementation, and evaluation of programs that support the University’s Cocurricular Learning Outcomes. • http://www.sa.psu.edu/hr/pdf/intorgchart.pdf

7. What does an assessment office do? • Conducts assessments, duh! • Provides training and professional development to others • Environmental scanning • Consulting • Brings together evidence • Contributes to strategic planning and accreditation activities

8. Assessment planning • There is a great deal of pressure to DO assessment, but where do you start?

9. First, review your goals Clues can be found in your: • Mission, vision, and objectives • Strategic plan • Stated learning outcomes Where do you want to be?

10. It is really helpful to get out of your silo • You can only learn so much from documents. • Institutional knowledge: • Who? • What? • How?

11. Next, develop a plan.

12. Start with a model II. Five Questions Are you meeting your goals? Are you improving? Do you meet prof. standards? How do you compare? Are you efforts cost-effective? IV. Collect & Analyze Evaluation Evidence V. Communicate Results Take Academic & Administrative Action Improve & Strengthen Programs 1. Janusian Duality For internal improvement For External accountability III. Select Methods & Measures For Academic Effectiveness: Student learning, Research, and Scholarship For Administrative Effectiveness: Strategic Planning & Resource Mgmt. For Each Level: -Institution -Unit -Individual students The Volkwein Model (2009)

13. The assessment cycle

14. Key steps - PDCA • Identify outcomes • Identify appropriate measures • Choose appropriate assessment methods • Choose an appropriate research design • Collect the data • Analyze & interpret the data • Disseminate the findings • Take action • Wash, Rinse, Repeat!

15. Think long-term • Chances are you cannot assess all of your outcomes every semester or even every year. • Be realistic about your resources and develop a comprehensive, multi-year plan. • Don’t forget to incorporate all steps of the cycle into your plan.

16. A good plan: • Collects multiple types of evidence (triangulation), • Takes into consideration your population(s) of interest, • Takes into consideration your audience, • Has a realistic timeline, • Identifies its products, • Incorporates dissemination & action, and • Is always subject to revision!

17. A matrix can be a helpful to organize your strategy and look for holes in your plan

18. Once you have your multi-year plan, you can bite it off in chunks. • Identify a subset of outcomes • Identify appropriate measures • Choose an appropriate assessment method • Choose an appropriate research design • Collect the data • Analyze & interpret the data • Disseminate the findings • Take action • Wash, Rinse, Repeat!

19. Your turn…

20. Example from the PRCC: • By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. • By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

21. Audience • By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures. • By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

22. Behavior • By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. • By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

23. Condition • By the end of the World Cultural workshop students will be able to describe four unique traditions, values and beliefs to other races and cultures. • By the end of the social justice retreat, students will demonstrate the ability to incorporate social justice techniques.

24. Next…

25. Determine appropriate measures • Attitudes? • Experiences? • Knowledge? • Abilities? • Retention? • Graduation? • Employment?

26. Some types of assessments • Standardized exams • Test of abilities or knowledge • Simulation or performance appraisals • Interviews and focus groups • External examiners • Surveys • Archival records and transcript analysis • Portfolios • Behavior observations • Student self-evaluations • Reflective writing • Minute papers/muddiest point

27. Tests • Can be used to measure a specific skill or knowledge base • “Objective” • Feasible? • Surveys • Attitudes • Experiences • Self-reported abilities & knowledge • Can be confidential/anonymous • Can reach large numbers of people

28. Interviews • Rich & detailed information • Body language can be helpful • Respondents may feel inhibited • Repeated interviews can build trust/comfort • Time consuming • Focus groups • Many of the same benefits as interviews, but not as in-depth • Participants “feed off of” each other • Can reach more people than interviews

29. Back to the PRCC: By the end of the World Cultural Workshop students will be able to describe four unique traditions, values and beliefs of other races and cultures. • Measure: Knowledge • Some possible assessment methods: • Knowledge test (direct evidence) • Survey self assessment of knowledge (indirect) • Portfolio • Reflective writing • Research design could be ex-post facto, pre-test/post-test, etc.

30. Next…

31. What if: • World Cultural Workshop participants rated their knowledge of other cultures more highly BEFORE the workshop than after (pre-/post-test design)? • In a reflective writing exercise you notice that participants are commonly mismatching the culture/race and its unique characteristics?

32. Next…

33. Potential refinements to the World Cultural Workshop based on our assessment • Have a single assessment in which students focus on what/how much they learned in the workshop, rather than a pre-/post-test comparison. • Cover a smaller number of races/cultures and spend more time on each. • Consider refining objectives

34. And again...

35. Questions? • If you are driving at the speed of light and you turn on your headlights, do they do anything? • How your boss is likely to make assessment plans:

36. Resources & References • Astin, A. W. (1993). Assessment for excellence: The philosophy and practice of assessment and evaluation in higher education. American Council on Education Series on Higher Education. Phoenix, AZ: Oryx Press. • Improving Educational Programming online training module: https://www.sa.psu.edu/workshops/edprogram/index.htm • Pope, R. L., Reynolds, A. L., Mueller, J. A., & Cheatham, H.E. (2004). Multicultural competence in student affairs. San Francisco: Jossey-Bass. • SARA Assessment Resources website:http://studentaffairs.psu.edu/assessment/resources.shtml • Volkwein, J. F. (2009). A model for Assessing Institutional Effectiveness. In J. F. Volkwein (ed.), Assessing Student Outcomes: Why, who, what, how?New Directions for Institutional Research Assessment Supplement (pp. 13-28). San Francisco: Jossey-Bass.