Leveraging Partnerships to Enhance Assessment Practices

1. Leveraging Partnerships to Enhance Assessment Practices Beth M. Lingren Clark, Ph.D. Director: Orientation & First-Year Programs University of Minnesota-Twin Cities

2. Outcomes • Evolution into assessment framework • Developing assessment plans • Involving key campus partners/stakeholders • Examples of using the data to increase the effectiveness of measures, themes and impact institutional projects

3. Old practices in assessment • Orientation & First-Year Programs (OFYP) discovered assessment methods only measured the level of satisfaction with our program/specific components. • OFYP asked questions unveiled that participants were satisfied to very satisfied. • Satisfaction not outcome orientated or as helpful when trying to enhance or improve your program. • OFYP aligned orientation program questionnaires. We worked with the Office of Measurement Services to help us identify common questions across all questionnaires.

4. Old practices in assessment • Shared data with presenters to enhance content • Shared with collegiate representatives to enhance college components • Influenced orientation program content and communications

5. Needed more information to identify content, make enhancements/improvements, inform partners to all populations served With this data…

6. Taking a Broader Approach

7. Think about what motivates a study(topic) and what shapes a study (question or problem).

8. “Research is a structured process of inquiry that draws upon existing knowledge and theory to advance general understanding or impact practices in the field at large.” -Henscheid, J.M.& Keup, J.R. (2011, p.ix)

9. Research: Topics/Questions • How does what we implement impact retention? • Is parent involvement important? • Why do students leave? • How are we impacting personal and academic success? • What key factors predict transfer student success?

10. Process: Topics/Questions • Who completes online Orientation? • Who attends on campus Orientation? • Who lives in Living Learning Communities • Who has completed the Strengths assessment? • What attended Transfer Welcome Day? • Which International students are coming?

11. Content: Topics/Questions • What are the gaps between expectations and reality of experiences? • How can we reduce anxieties/concerns? • How can we increase parent involvement? • How do we create a sense of belonging?

12. Evolution to Assessment Framework

13. Where we started… Welcome Week Committee Structure Purpose of Committee: (Continued) • Identify existing instruments to collaborate data collection • Discuss implementation of the program evaluation • Discuss attendance tracking at events Representatives from the following areas: Provost’s office, Orientation & First-Year Programs, Office of Measurement Services, Institutional Research, College of Liberal Arts, Student Activities, Center for Teaching and Learning, Department of Educational Policy & Administration, Career Center, Student Leadership & Engagement

14. The process • Followed cohorts from old program – New Student Weekend (2001 and 2002 cohorts) • Welcome Week planning occurred simultaneously with the development of the First Year Assessment Plan (baseline cohort 2007) • Did not want to implement WW in isolation • Support the efficacy of WW • Created a mapping processing and aligned with what mattered at the institution • Identified what else existed system-wide to support first year content and themes

15. Assessment Cycle Identify learning objectives/goals for program Map learning objectives/program goals to Student Learning and Development Outcomes Conduct gap analysis Change/enhance content based upon gap analysis and program survey results from previous year (Common questions) Implement program introducing objectives and outcomes to population Evaluate program Analyze and share results Repeat

16. How Do We Create a Common Language of Assessment? Understanding the assessment framework • Goal: Initiating undergraduates into Transformative Education • Student learning & development outcomes • Two primary transformations… • Transforming the campus into the curriculum • Transforming the undergraduate experience • Assessing first-year programs • Have goals we care about • Know what you know • Map to goals

17. How Do We Create a Common Language of Assessment? Mapping Outcomes • Mapping process is a conversation • Meant to be iterative; refined over time • Each step informs the next • What we needed to begin mapping • Schedule and description of goals/sessions/events • List of student learning and development outcomes • Mapping spreadsheet (essentially a matrix of sessions/outcomes)

18. How Do We Create a Common Language of Assessment? Mapping Outcomes • Was done on a large scale (for all of Welcome Week’s content) • Mapping also done on a smaller scale • College Day - College of Liberal Arts

19. The Power of the Common Question • We wanted to: • Create a common language beyond orientation programming • Create an intentional plan for content and evaluative measures • Implement transformational practices on campus • “We wanted to speak a common language”

20. The Power of the Common Question (continued) • We wanted to: • Create a profile of people completing our program questionnaires • Create longitudinal data from one year to the next as well as within the cohort/year • Ask more specific questions, AND ask participants to elaborate on items where they may not have been satisfied • Understand the utility of common mapping

21. Development of a Formal First-Year Assessment Plan

22. First-Year Assessment Plan

23. Intent of FYA Plan • The intent of this process provides data on student behavior related to: • Matriculation process • Student participation • Student satisfaction • Student success both personally and academically • Longitudinal data utilized • Measure the impact on student retention and other programs • The plan helps to inform institution on the things we were learning about from our students and their experiences

24. Involving Campus Partners • Office of Measurement Services • Undergraduate Education • Collegiate Units • Academic Support Resources • Admissions, Housing, Athletics, Honors, International • Office of Institutional Research • Strengths • Office for Student Affairs

25. Examples of Using the Data

26. Matriculation • Tracking… • Online Orientation completion • Orientation attendance • Document check for international • Parent attendance • Welcome Week/Transfer Welcome Day attendance • Strengths completion

27. Participation

28. Survey Data • Pre-Orientation Survey • Expectations/Welcome messages • Orientation content (student and parent) • Common Questions • Cross college • Theme/Program content

29. Survey Data • Program Evaluations • Improvement • Outcomes based • Check-In Surveys • Reality of expectations • Additional support and resources • Newsletter content

30. Institutional Data • National Survey on Student Engagement • Comparing freshmen and transfer student experiences • Student Experience in a Research University • Welcome Week and Sense of Belonging • Transfer Research Grant results

31. Institutional Data • Leaver’s Analysis • Longitudinal data/Collegiate specific • Importance of Parent/Family Engagement • @ Home in MN • Student Retention • Transfer Grant • Influence overall experience and engagement

32. Additional Themes… • Interactions with faculty are an important part of the college experience. • The majority of the students spend between 11-20 hours attending class per week. • On average, students spend up to 10 hours per week on social networking (facebook). • Transfer students have been successful adjusting to the academic demands and developing effective study skills but have had more of a difficult time getting to know their peers and their faculty

33. Additional Themes… • Student’s top major anxiety/concerns: • academic coursework • making friends • financial concerns • deciding on a major • developing skills related to note-taking/studying • time management • stress management • transfer of credits**

34. Additional Themes… • The University can feel like a big place so it is important for students to create community. • Students can do this in their residence hall • Through the Commuter Connection program • In class • At work • Joining a student organization. • Especially important for students who are not from the Twin Cities area.

35. Additional Themes… • New students leave the U for various reasons: • Financial concerns • Academic • Medical • Family • Other personal issues

36. Additional Themes… • At the six to eight week mark, students are: • seeking more contact with advisers • want help with study skills • more opportunities to socialize** • information on campus libraries** • more interactions with faculty**

37. Sharing the Data • College/unit level data is reported to committees and administrators • Mapping Process revisited to ensure meeting goals and desired programmatic outcomes and University Student Development and Learning Outcomes

38. Sharing the Data • Program evaluation data will be distributed to content creators and future planning committees • Data presented to the students (e.g., Advice from the Class of 2016/Transfer?) • Annual reports disseminated to leadership and decision-makers within Undergraduate Education

39. Where Do We Go From Here? • Utilize the Data • Use the findings for continuous improvement efforts • Did we do what we set out to do? • What can we do better? • What was successful and should be continued? • How do we alter content to address new issues? • Compare findings year to year • Are students attitudes/perceptions changing? • Are there new transformational issues? • Longitudinal Analysis • Has Welcome Week impacted the student experience? • Are there correlations between student attendance, satisfaction retention, graduation rates, etc.? (Utilize orientation database)

40. Inspiring Our Work • Influence program content, delivery, and messaging • Integrate data results directly into what we share with students and families • Share with colleges and units to influence the way we do our work • Strengthens partnerships • Provide evidence to work with units to make strategic changes

41. “Educators who work to improve the experience of college students, especially students facing transitional periods, are typically driven by passion for the work. At their best, these educators combine this passion with a solid understanding of how students experience college, what helps or hinders the students’ progress, and what outcomes result from their experience. -Henscheid, J.M.& Keup, J.R. (2011, p.17)

42. Resources http://www.ofyp.umn.edu/en/welcome-week.html http://www.ofyp.umn.edu/en/survey-results.html http://www.ofyp.umn.edu/ofypmedia/pdfs/highered/reports/2011_OFYPAnnual.pdf http://www.oir.umn.edu/ http://www.oir.umn.edu/surveys/seru

43. Reference Henscheid, J.M. & Keup, J.R. (2011). Crafting and conducting research on student transitions. Columbia, SC: University of South Carolina, National Resource Center for the First-Year Experience and Students in Transition.

44. “When done well, quantitative research has the potential to enhance knowledge and inform programs, policies and pedagogies that educators use to serve students and contribute to their success” (Henscheid & Keup, 2011, p.84)

45. Questions/Comments Beth LingrenClark, Ph.D. Director: Orientation & First-Year Programs lingr004@umn.edu