Who Are Our Students??? Assessment Forum

1. Who Are Our Students???Assessment Forum Monday, Feb. 9, 2004 Donald J. Lerew, Jr. – Institutional Researcher Ronald Burwell – Professor of Sociology

2. Overview • Welcome – Review of Agenda • IR & Assessment Website & Data Sets • Assessment Calendar • AICUP First-Year Student Survey 2003 • Senior Interviews – Some Highlights • NSSE Student Engagement Survey 2002 • CSS Senior Survey 2003

3. Talking Points • How do seniors view their time at Messiah? • How many hours do our students spend in academic pursuits in a given week? • How do students’ views change regarding gender roles? • Who stays and who leaves…and why? Social vs. Academic Factors? • How much effort do our students put into their college experience?

4. Research/Assessment Web Site • General, college-wide statistics • IR Publications Listing • Assessment Calendar • Assessment data • CIRP, CSS, NSSE, CSEQ, SSI, FYSS • Location: Messiah Home > Campus Offices > Institutional Research

5. Survey Data Sets • Most analyses have datasets available • Datasets typically created for SPSS • Contact Ron or I for a copy

6. Assessment Calendar • Schedule of planned institutional assessment activities over next several years • Presented and reviewed by Assessment Committee • Goal is to not overburden students, while gathering sufficient, meaningful data • Calendar is available on IR website

8. Assessment Calendar • CIRP – Cooperative Inst. Research Program • Survey of New Freshmen at orientation • Baseline picture of who students are before college • AICUP First-Year Student Survey • Survey of first-year students in spring semester • Assesses academic and personal development of students during their first year of college

9. Assessment Calendar • NSSE – National Survey of Student Engagement • Given to 2nd-semester freshmen and seniors • Studies quality of UG learning on campus by way of student engagement • CSS – College Student Survey • Surveys Messiah seniors • Students’ self-reported academic and campus life experiences

10. Assessment Calendar • CSEQ – College Student Experiences Questionnaire • Surveys seniors • Assesses the quality of student effort in utilizing resources/opportunities at college • SSI – Student Satisfaction Inventory • Surveys subset of each class level • Measures student importance and satisfaction to various campus issues

11. Assessment CalendarOther Possible Instruments • YFCY – Your First College Year • Supplement to CIRP • FSSE – Faculty Survey of Student Engagement • Others???

12. AICUP – First-Year Student SurveySpring 2003 • Survey of first-year students • Surveyed during spring semester • Surveyed through campus mail • General and retention analyses conducted

13. AICUP – First-Year Student Survey--Self-Assessment-- • MC men are significantly less prepared & are significantly more challenged in the area of study skills during the first year • MC students indicate study and writing skills as their two biggest first-year skill challenges • Both self-rating and first-year assessment of spirituality traits were significantly better than comparison institutions

14. AICUP – First-Year Student Survey--Transition to College-- • MC women experience significantly more homesickness than men • MC students more likely to be homesick than at comparison institutions • MC men report significantly less perceived help in areas of: • Dealing with roommate problems • Dealing with personal problems • Working with advisor

15. AICUP – First-Year Student Survey--Academic Advising-- • MC men relied significantly more on themselves for academic advice than did women (M-28.6%, W-15.4%) • MC students placed more emphasis on soliciting academic advice from family (MC-25.9%, Comp-15.1%) • MC students placed less reliance on the advisor (MC-21.4%, Comp-29.0%)

16. AICUP – First-Year Student Survey--Academic Advising-- • MC women met with advisor more often during first year (M-2.55, W-3.26 mtgs.) • Comparison group averaged 3.80 meetings • MC students rated advisors significantly better than comparison group students in: • Knowledge level, availability, & overall satisfaction

17. AICUP – First-Year Student Survey--Use of Time-- • MC students spend nearly 22 hrs/wk attending class or studying/doing homework • MC men spend significantly more time than women watching TV and playing video/computer games • MC women spend significantly more time in prayer and meditation

18. AICUP – First-Year Student Survey--Use of Time-- • MC students spend significantly more time than comparison group students in: • Prayer/meditation • Studying and completing homework • Participating in student clubs/groups

19. AICUP – First-Year Student Survey--Satisfaction-- • MC women significantly more satisfied in: • Quality of academic advising, quality of academic offerings, availability of academic offerings, Internet access, faculty attitude toward students, and personal counseling • MC students indicated significantly higher satisfaction than comparison students in: • Library facilities/resources, food services

20. AICUP – First-Year Student Survey--Satisfaction-- • 93.7% of MC students indicated that they would “definitely” or “probably” return for the second year (Comp. Group - 90.2%) • Prominent reasons for not returning • Transferring • Dissatisfaction with social atmosphere • Lack of financial resources

21. AICUP – First-Year Student Survey--Retention Analysis: General-- • Attrited students reported significantly… • Higher levels of homesickness and loneliness • Lower levels of belongingness • Attrited students indicated significantly less college help in the areas of: • Personal problems, student club involvement, organized social events, rec. athletic activities

22. AICUP – First-Year Student Survey--Retention: Skills Assessment-- • Attrited students rated themselves significantly lower in the areas of: • Intellectual self-confidence, social self-confidence, ability to function independently, leadership ability, reading analytically, and spirituality • While incoming skill levels differed, no significant differences in reported growth during first year

23. AICUP – First-Year Student Survey--Retention: Other Items-- • Attrited students were significantly more likely to indicated the advisor as the primary source of academic advice; equally less on self-reliance • Retained students spent more time socializing with friends (7 vs. 4 hrs/wk) • Both student groups indicated similar academic, residence hall, and social life expectations • Attrited students indicated significantly lower levels of having those expectations met during the first year

24. Senior Interviews – Spring 2002 • Approximately 27 completed interviews with seniors during the spring of 2002 • Transcriptions created for 24 interviews (sound quality created some difficulties in transcription) • Topics: Educational Experiences; Religion; Gender

25. Educational Experiences • Based on questions from Richard Light • Academic regrets • Unforgettable experiences • Critical moments • Important books • Interactions in the residence hall

26. Religion • Awareness of and introduction to college’s religious heritage [related to Lilly Project] • Important formative faith experiences • Experiences with doubt • Understanding of what is a “mature Christian” [James Fowler]

27. Gender • Views on male / female roles with society and the family • Views on ordination of women and role in the church • Impact of Messiah College on ideas regarding gender

28. Senior Voices • Academic regrets • Unforgettable experiences • Residence Hall interactions • Views on gender roles • Ordination of women

29. NSSE – Student EngagementSpring 2002 • Survey of 1st-yr. and senior students • Determine extent to which students engage in educational practices that are associated with learning outcomes • Surveyed during spring semester • Web administration • General and school analyses

30. NSSE – Student Engagement

31. NSSE – Student Engagement--Acad/Intellectual Experiences-- • Only 21% of MC seniors worked with other students on projects during class “often” or “very often” (Bac-Gen & NSSE – 43%) • MC first-year students worked significantly more often with classmates outside of class to complete class assignments • MC students discussed class ideas with others outside of class significantly more often than comparison students

32. NSSE – Student Engagement--Acad/Intellectual Experiences-- • MC women were more likely than men to: • Use e-mail to communicate with instructor • Work harder than they thought they could • Discuss ideas with others outside of class • Work with classmates outside of class • Students w/ a “B- or lower” average were more likely to: • Come to class unprepared • Work with others on projects during class

33. NSSE – Student Engagement--Mental Activities-- • Five Levels: (1) Memorize, (2) Analyze, (3) Synthesize, (4) Make Judgments, (5) Apply • MC students had similar category breakdowns to comparison students • Exception: both first-year and senior MC students indicated lesser levels of memorization • MC students w/ “B- or lower” indicated significantly higher usage of memorization

34. NSSE – Student Engagement--Reading & Writing-- • MC students indicated a significantly higher number of assigned textbooks/readings than comparison students • MC students indicated writing significantly more 5-19 and under-5 page papers

35. NSSE – Student Engagement--Educational Experiences-- • MC students indicated significantly higher participation/plan to participate rates in: • Practicums, internships, or field experiences • Community service or volunteer work • Foreign language coursework • Study abroad • Culminating senior experience (exam, capstone, project, etc.) • MC women are more likely to attend

36. NSSE – Student Engagement--Time Usage-- • MC students spend significantly more time preparing for class than do comparison students (FY-14.5, Sr.-14 hrs/wk) • MC FY students spend 11 hrs/wk relaxing/socializing; 9 hrs/wk for seniors • MC students work significantly more on-campus, but significantly less off-campus

37. NSSE – Student Engagement--Time Usage-- • MC first-year women spend significantly more time preparing for class; senior men, however, spend more time than women • MC students with a “B- or lower” spend: • Less time preparing for class (9 vs. 14 hrs/wk)

38. NSSE – Student Engagement--Educational/Personal Growth-- • MC students indicated significantly more growth in: • Acquiring a broad general education • Developing personal code of values/ethics • Contributing to welfare of community • Writing clearly and effectively • Thinking critically and analytically • MC first-year students reported significantly less growth in the use of computer/info. technology than did comparison students

39. NSSE – Student Engagement--Institutional Emphasis-- • MC students reported significantly more emphasis by the college on: • Spending significantly more time studying and on academic work (seniors only) • Encouraging diversity contacts/interaction • Providing support needed to thrive socially • Attending campus events and activities • Messiah women indicated higher perceived emphasis toward academic and social support

40. NSSE – Student Engagement--Satisfaction-- • Over 90% of MC men and 95% of women rated the educational experience as “good” or “excellent” • Significantly higher than comparison groups • A high majority of students indicated they would “probably” or “definitely” return • FY (W-94.7%, M-90.6%) • Sr. (W-93.8%, M-84.6%) • Students w/ “B- or lower” evaluated educational experience significantly lower than “B or better”

41. NSSE – Student Engagement--Analysis by School-- • Summarizes mean responses by school • Compares each school mean by the aggregate of all other schools • Compares each school mean by class level

42. College Student Survey - 2003

43. Sample • Sample size – 267 graduating senior • 32.2% male • 67.8% female • Slight oversample of females

44. Plans for fall 2003 • Approximately 1 in 5 seniors plan on graduate study in the fall • Most of the remaining students plan on working either part time or full time

45. Academic Activities • MC students report frequent peer interaction in connection with academic activities (e.g., studying with other students and discussing courses with other students) • MC students were more likely to be guests in professors homes compared to other private colleges

46. Other Activities • Some 38.7% report drinking beer during the last year • 56.8% report consuming wine or alcohol during the past year • While these are lower than comparable rates at private colleges they are higher than in 2001 • About 40% of MC students begin to use alcohol while in college

47. Other experiences • A majority of students reported that they occasionally or frequently were homesick (64.7%) or depressed (63.9%) • About one third of the students reported frequently feeling overwhelmed (38.7%)

48. Time Usage • MC students report that they spend about 25 hours per week on academic pursuits (classes/studying) • A comparison sample from private colleges report about 22 hours per week on the same activities • In 2001 MC students reported slightly more time spent on academic pursuits (28 hours)

49. Interactions with Professors • MC students report that professors frequently provided intellectual challenge (55.6%) and emotional support (36.5%)

50. Satisfaction with college • Some 91% of MC students are satisfied with their overall college experience (private – 86%) • Most MC students say they would choose the same college again (MC – 89%, private – 80%)