Developing and Assessing Outcomes in the Co-Curricular Experience

1. Developing and Assessing Outcomes in the Co-Curricular Experience Prepared for Auburn University’s Division of Student Affairs A-Team April 2, 2013 Sandi Osters, Ph.D. Retired Director, Student Life Studies Texas A&M University 3001 Aztec Street College Station, TX 77845 sandiosters@gmail.com

2. Workshop Overview • Student integrative learning • Outcomes-based assessment • Learning outcomes – the heart of the matter • Working sessions • Assessment methods • Working sessions • Reflective dialogue

3. Today’s Program Outcomes • The presenter will: • Work on issues important to participants • Provide a general understanding about the importance of outcomes based assessment in student affairs • Emphasize the importance of alignment with division and university mission, goals and planning • Emphasize the use of data to inform practice

4. Today’s Learning Outcomes • Participants will: • Describe learning and program outcomes for their own program, office or activity • Employ assessment methods for their outcomes • Develop initial action steps • Articulate what they have learned and what they still need to know

5. Handouts • PowerPoint • Writing Outcomes • Assessment Plan Template • Rubrics • Everything You Ever Wanted to Know about Assessment – on One Page

6. It’s All About Students’ Integrative Learning • A strong emphasis on student learning is the primary key to retention through graduation • Learning is the primary activity and goal of the college environment – both inside and outside of the classroom • The integration of academic programs and co-curricular activities is important • The ability of students to integrate their learning both inside and outside of the classroom – be responsible for their own learning – is of the greatest importance

7. What Is Student Learning in the Co-curricular? “Learning Reconsidered defines learning as a comprehensive, holistic, transformative activity that integrates academic learning and student development, processes that have often been considered separate, and even independent of each other.” “Student learning produces both educational and developmental outcomes; distinguishing them is pointless and potentially harmful, and the goal of higher education should be the integration of all domains of learning and of the work of all educators.” Keeling, R. P. (Ed.) (2004). Learning Reconsidered. Washington, D.C.: NASPA and ACPA.

8. What is Assessment and Evaluation? “Assessment is any effort to gather, analyze, and interpret evidence which describes institutional, departmental, divisional or agency effectiveness.” “Evaluation is any effort to use assessment evidence to improve institutional, departmental, divisional, or agency effectiveness.” Upcraft, M. & Schuh, J. (1996). Assessment in student affairs: A guide for practitioners. San Francisco: Jossey-Bass.

9. The Purpose of Outcomes Based Assessment • To answer some simple questions: • What is it that we do? • Why do we do it? • How well do we do it? • How do we know that we do it well? • What improvements and changes do we make from what we learn? • Do the improvements work? • To genuinely engage faculty, staff and students in the day-to-day reflection of answering these simple questions • To demonstrate a commitment to systematic examination of the quality of all wedo to improve our programs and departments

10. The Purpose of Outcomes Based Assessment, continued • To celebrate our successes and learn from our failures • To inform our policy, programs, processes and resource allocations • Increase our confidence that we are putting our time and energy into activities that results in the outcomes we and our students value • To have the data to support why our programs generate the learning Auburn wants for its students in times of financial downsizing • To be able to articulate exactly how student affairs professionals contribute to student learning and student development • To gather and display data that will allow us to satisfy accrediting agencies and program reviews (CAS or your own)

11. Advantages of Using Outcomes Language in Student Affairs • Make it clear to students who participate in our programs, activities and services what they can expect to gain from doing so • Make it clear to others what the program will accomplish and, where appropriate, what students will learn • Help staff select appropriate strategies and assessment measures/methods to reach the outcomes

12. Advantages of Using Outcomes Language in Student Affairs • Apply the results of our assessments for meaningful improvement and/or change • Move beyond student satisfaction and tracking the use of services as the sole means of describing student affairs effectiveness • Connect Student Affairs to the academic enterprise – we all are about student learning – we all are educators

13. What Any Assessment is NOT • Assessment does not exist for assessment’s sake • It informs but does not drive where you want to go and what you want to be as an office, activity, program • It is not used for personnel evaluation • It is not a one time thing – but systematic and iterative • It does not assess all things, all the time

14. Outcomes Based Assessment and Decision Making • Most importantly, Outcomes based decision making and assessment should be • Understood = by faculty/professionals and students • Meaningful = faculty/professional (i.e., expert) driven • Inclusive = involve as many faculty, student affairs professionals and students as possible • Manageable = takes into account varying resources • Flexible = takes into account assessment learning curves • Truth-seeking/objective/ethical • Iterative and systematic • Inform decisions for continuous improvement or provides evidence of proof • Promote a culture of accountability, of learning, and of improvement Bresciani, J. J., Zelna, C. L., & Anderson, J. A. (2004). Techniques for assessing student learning and development: A handbook for practitioners. Washington, D.C.: NASPA, Inc.

15. Assessment Cycle Gather Data Interpret Evidence Mission/Purpose Goals Objectives/Outcomes Environment Budget Implement Methods to Deliver Outcomes and Methods to Gather Data Politics Make decisions to improve programs; enhance student learning and development; inform institutional decision- making, planning, budgeting, policy, public accountability Scholarship Accountability

16. Writing Outcomes • Outcomes – Detailed, specific, measurable or identifiable, and personally meaningful statements that are derived from goals and articulate what the end result of a unit, program, course, activity, or process is. • Learning Outcomes – An easily identified action that a student is expected to demonstrate in terms of knowledge, skills, and attitudes upon completion of a program/activity/event. • Program/process outcomes – The end result of what a program or process is to do, achieve, or accomplish. • Write simply and with one outcome per statement

17. Bloom’s Taxonomy (1956) A structure for developing learning outcomes

18. Learning Outcomes? • Students will be satisfied with the information received. • Students will rate the service received positively. • Students will create their own personal leadership philosophy based on three models taught in Lead 101 in the fall semester • After the service project, students will be able to articulate the root cause of homelessness in Auburn, Alabama.

19. Disability Services Example • Learning outcome • Students will be able to describe their functional limitations in an academic setting • Assessment method • Pre-test/post-test (intake form) • Rubric • Criteria for success/achievement target • Students post-scores will improve over pre-test

20. Disability Services Example • Learning outcome • Students will be able to describe how their disability may affect them in the workplace and what accommodations they may need to ask for from their employers • Assessment method • Exit interviews with graduating students • Criteria for success/achievement target • 80% of students in their graduating semester will be able to identify at least one way their disability might effect them in the workplace.

21. Program/Process Outcomes? • Students will like the conference banquet • The professional development sub-committee will offer two dining etiquette programs in the fall semester for graduating students • Develop and expand the Student Health Services web page in order to increase student access to health information, patient services and educational programming

22. Examples of Program Outcomes • The Department of Residence Life will be able to respond to maintenance requests within 24 hours of their walk-in time • The Student Health Service will be able to admit students for check-ups within 24 hours of their walk-in time. • The Financial Aid Office will have full award letters out to the on-time complete applicants by April 15th.

23. Program Outcome Example • Program outcome • Develop and expand the Student Health Services webpage in order to increase student access to health information, patient services, educational programming • Assessment methods • Data retrieval from webpage interactions • Annual patient survey to determine student utilization of the web page • Survey instruments by Health Education program to determine student utilization of web page • Criteria for success/achievement target • Increase in the utilization of the SHS webpage regarding health information, patient services and educational programming.

24. Common Mistakes in Writing Learning Outcomes • The learning outcomes don’t align with department, division or university goals • Outcomes include words that are hard or impossible to measure (understand, appreciate, know about, become familiar with, learn about, become aware of) • Outcomes include too many skills in one statement • Outcomes measure satisfaction or performance evaluation rather than learning of the student • There are too many learning outcomes • Only one person wrote, reviewed, edited and implemented the outcome

25. Working Session • Work in small groups of two or three • Develop at least two learning outcomes and two program outcomes for the Division of Student Affairs Global Service Initiative (mission provided) • Don’t worry about being an expert on this initiative. Pretend you are. What’s important is working on the outcomes • Share time

26. Assessment Methods • Qualitative and quantitative • One time and longitudinal • Direct - requires students to display their knowledge, behavior or thought processes • Indirect – asks students to reflect upon their knowledge, behaviors or though processes • Local (Auburn developed) and national assessments (like the NSSE or CIRP)

27. Direct Measures • Benefits of direct measures to Student Affairs • Provides faculty and administrators more confidence in the measures and their results because it is the academic standard • Helps with accreditors who are looking for student learning in the co-curricular and measures of same • Challenges • Time and expertise needed to develop • Assessing the “fuzzies” – attitudes and values • Small “n’s” and the validity of the studies • Bottom line • Use multiple indirect measures to offset the lack of direct measures

28. Common (and still viable) Assessment Measures • Surveys to inform of needs and satisfaction • Surveys to assess outcomes • Focus groups/interviews to inform future assessments or to dig more deeply into existing survey results • Tracking use and participation (numbers)

29. New Wave of Assessment Measures • Exit Interviews • Learning contracts • Photography • Pre and post tests of knowledge • Observations with documentation • One minute papers • Rubrics • Reflective journals • Reflective conversations/writing • Combination of the above

30. Working Session • Working individually, develop at least two learning outcomes for a program or activity of interest to you. If you do not work directly with student learning, consider learning outcomes for your student workers • Develop at least one assessment measure for each outcome and address how you will know if you have been successful (criteria for success/achievement target) • Work in small groups and share your outcomes and measures. • As a group, develop an action plan (see Assessment Plan Template, Implementation of Assessment Process) for one of the learning outcomes in your group • Share

31. Sharing Your Assessment Data • Have clear objectives/purpose • Know who cares about your results • Effectively analyze the data • Effectively report/present results • Determine the best format – written reports, websites, newsletters, publications, oral presentations • Use the data yourself – role model to others • Share with each other – assessment road shows/poster sessions at division meetings/monthly during the A-Team meetings • Share with students!! What actions have you taken from what you have learned from them?

32. In Summation • Be clear about who you are, why you do what you do, how well you do what you do, and how you improve and change • Check in annually to be sure your program/activity and event outcomes align with your departmental mission/goals and that your departmental mission aligns with the division’s mission/goals and strategic plan and that your division mission/goals align with institutional mission/goals/strategic plan/QEP • Be patient; you will get better at writing outcomes; you always will have more to learn! • Practice, practice, practice and learn by teaching others • Work with your students and student groups to articulate outcomes for their programs • Work with colleagues, faculty and others with expertise • Celebrate your accomplishments

33. Contact Information Sandi Osters, Ph.D. Retired Director, Student Life Studies Texas A&M University 3001 Aztec Street College Station, TX 77845 sandiosters@gmail.com