Measuring and Managing Student Engagement: Why it Matters in the First Year of College Jillian Kinzie Indiana Universit

1. Measuring and Managing Student Engagement: Why it Matters in the First Year of College Jillian Kinzie Indiana University Center for Postsecondary Research Wisconsin OIPD Spring Conference 2007 Partnering for Student Success in the First Year: Learning About Learning

2. Current Concerns about College & Student Learning Colleges and universities, for all the benefits they bring, accomplish far less for their students than they should. -- Derek Bok At a time when the quality of postsecondary education seems to be slipping, participating in engaged learning activities promises to prepare students for a lifetime of continuous learning so that they and the country stay competitive in the global marketplace -- George Kuh The first year presents a unique opportunity to engage students in the habits of learning. -- John Gardner

3. College-going stakes higher today than at any point in history 45% students in 2yr-colleges depart during their first year, & 1 of 4 leave from 4-yr schools 51% of high school grads have reading skills necessary for college; 25% of students in 4-yr colleges need 1 yr of remedial coursework Enrollment & persistence rates of historically underserved students lagging Students and the College Experience

4. What We Know About the Undergraduate Experience from NSSE: • Full-time students spend an average of 13 hrs per week studying (Hrs. recommended by faculty = 20-25) • 45% of all college seniors took at least one course at another institution prior to enrolling at their current school – “swirling” • Three of ten first-year students reported working just hard enough to get by. • Between 40% and 50% of first-year students never used career planning, financial advising, or academic tutoring services. • 45% first-years and 30% seniors never discussed ideas with faculty outside class • Faculty spend 42% of class time lecturing (FSSE)

5. Current Context for Concern Engaged pedagogy is really the only type of teaching that truly generates excitement in the classroom that enables students and professors to feel the joy of learning -- bell hooks Conclusion: The first year experience is not being used as effectively as it could to engage students at high levels to develop habits of learning that insure student success

6. Principles of Good Practice for the First College Year: Recommendations • Institutional commitment by leaders, faculty, staff, and governing boards • Focus on student learning both inside & outside the classroom • Encourage student affairs-academic affairs partnership • Offer challenge and support • Communicate high expectations • Foster an inclusive and supportive campus climate • Conduct systematic assessment • Create an atmosphere of dignity and respect for first-year students • Teach students strategies and skills to succeed • Get faculty involved • Encourage students to assume responsibility for their success Challenging and Supporting the First-Year Student: A Handbook for Improving the First Year of College By M. Lee Upcraft, John N. Gardner, Betsy O. Barefoot, and Associates, Jossey-Bass, 2005 Pg. 515-517

7. To Ponder: • What data do you use to inform your understanding of the first year experience? In teaching first year students? • In what curricular and co-curricular areas, specific courses, or transition places do first year students have the most difficulty? What measures do you use to monitor these issues? • What are your current strengths in the first year experience?

8. Student Success in College Student success - defined in a broad, all-encompassing manner includes: academic achievement, engagement in educationally purposeful activities, satisfaction, acquisition of desired knowledge, skills and competencies, persistence, attainment of educational objectives including graduation, and post-college performance. http://nces.ed.gov/npec/pdf/Kuh_Team_Report.pdf

9. • Study Habits • Peer Involvement • Interaction with Faculty • Time on Task • Motivation • Other • First Year Experience • Academic Support • Campus Environment • Time on Task • Peer Support • Teaching & Learning Approaches • Other Pre-college experiences

10. What Really Matters in College: Student Engagement The greatest impact appears to stem from students’ total level of campus engagement, particularly when academic, interpersonal, and extracurricular involvements are mutually reinforcing… Pascarella & Terenzini, How College Affects Students, 2005, p. 647

11. What Really Matters in College: Student Engagement Because individual effort and involvement are the critical determinants of college impact, institutions should focus on the ways they can shape their academic, interpersonal, and extracurricular offerings to encourage student engagement. Pascarella & Terenzini, How College Affects Students, 2005, p. 602

12. Student Success: Lessons From the Research • What matters most is what students do, not who they are • A key factor is the quality of effort students expend • Educationally effective institutions channel student energy toward the right activities

13. Two Components of Student Engagement 1. What studentsdo -- time and energy devoted to educationally purposeful activities 2. What institutionsdo -- using effective educational practices to induce students to do the right things

14. Foundations of Student Engagement Time on task (Tyler, 1930s) Quality of effort (Pace, 1960-70s) Student involvement (Astin, 1984) Social, academic integration (Tinto,1987, 1993) Good practices in undergraduate education (Chickering & Gamson, 1987) College impact (Pascarella, 1985) Student engagement (Kuh, 1991, 2005)

15. Good Practices in Undergraduate Education(Chickering & Gamson, 1987; Pascarella & Terenzini, 2005) • Student-faculty contact • Active learning • Prompt feedback • Time on task • High expectations • Respect for diverse learning styles • Cooperation among students

16. Lessons from Research on College Impact Effective Educational Practices for Fostering Engagement: • Academic Challenge • Active and Collaborative Learning • Student-Faculty Interaction • Enriching Educational Experiences • Supportive Campus Environments

17. Promise ofStudent Engagement “If faculty and administrators use principles of good practice to arrange the curriculum and other aspects of the college experience, students would… write more papers, read more books, meet with faculty and peers, and use information technology appropriately, all of which would result in greater gains in such areas as critical thinking, problem solving, effective communication, and responsible citizenship.” Kuh, Kinzie, Schuh, Whitt & Associates, Student Success in College, 2005

18. National Survey of Student Engagement(pronounced “nessie”)Community College Survey of Student Engagement(pronounced “cessie”) Undergraduate student surveys that assess the extent to which students engage in educational practices associated with high levels of learning and development

19. NSSE Project Scope • One million+ students from 1,100 different schools • Random sample of First-year and Seniors surveyed spring term • 50 states, Puerto Rico, Canada • 70+ consortia WISCONSIN SYSTEM! Related Initiatives: • Faculty Survey of Student Engagement (FSSE) • Beginning College Survey of Student Engagement (BCSSE)

20. NSSE sightings all over Wisconsin!

21. Taking a look at NSSE: • Based on effective educational practices • Designed and tested for high validity and reliability • Relatively stable over time • High credibility of self-reported data • Students will participate • Actionable data • Psychometric properties document on Web site

22. NSSE Survey Student Behaviors Student Learning & Development Institutional Actions & Requirements Reactions to People & Environment Student Background Information

23. Q.1 – Academic activities Q.2 – Learning mental activities Q.3 – Reading & writing Q.4 – Homework Q.5 – Academic challenge Q.6 – Co-curricular activities Q.7 – Enriching educational experiences Q.8 – Campus relationship Q.9 – Time usage Q.10 – Institutional emphasis Q. 11 – Gains Q.12 – 14 – Satisfactions NSSE Survey

24. NSSE’s Five Clusters ofEffective Educational Practice Supportive Campus Environment Enriching Educational Experiences Level of Academic Challenge Active & Collaborative Learning Student Faculty Interaction

25. NSSE Results A window into the undergraduate experience Discover strengths and weaknesses in educational program Identify areas that need attention to improve student learning and success Help pinpoint aspects not in line with mission, or what institution expects Link with other institutional data

26. What can we learn about first year student engagement from NSSE?

27. What do first-year students do? 1. What percent of full-time first-year students study, on average, more than 20 hours per week? (a) 12% (b) 18% (c) 30% (d) 41% (e) 50% b. 18% NSSE FY; Wisconsin FY= 16%

28. Worrisome Gap? Time spent studying • First-year students average about 13-14 hrs. per week studying • Faculty Survey of Student Engagement (FSSE) data indicate that faculty expect students to spend more than twice that amount preparing (estimated 24-30 hrs. a week for FT) • Entering first-year students EXPECT to study more than they actually do in college

29. What do first-year students do? 2. What percent of first-year students frequently (“very often” + “often”) asked questions in class or contributed to class discussions? (a) 28% (b) 35% (c) 47% (d) 58% (e) 65% d. 58% NSSE FY; Wisconsin = 51% WI Range = 31% - 57%

30. First Year Students – Ask Questions in Class

31. Active and Collaborative Learning Scales • Collaborative Learning Scale • Worked with peers on projects during class • Worked with classmates outside of class to prepare assignments • Tutored/Taught other students • Discussed ideas from readings or classes with others outside of class (students, family) • Active Learning Scale • Asked questions in class • Make a class presentation • Participated in a community-based project (service learning) as part of a regular course

32. What do first-year students do? 3. True or False. Almost half of all students spend no time on co-curricular activities. True. 43% NSSE first-years, 48% seniors report 0 hours

33. FY Students – Time Spent on Cocurriculars

34. What do first-year students do? 4. What percent of first year students report they frequently (“often or very often”) received prompt feedback on their academic performance? (a) 27% (b) 35% (c) 44% (d) 53% (e) none of the above d. 53% NSSE first-years, Wisconsin = 48%

35. FSSE & NSSE comparison Prompt Feedback Lower Division Upper Division FACULTY gave prompt feedback often or very often 93% | 93% 1st yr. Students Seniors STUDENTS received prompt feedback often or very often 64% | 76%

36. What do first-year students do? 5. What percent of first-years frequently* prepared 2+ drafts of a paper or assignment before turning it in? (* % very often and often) (a) 65% (b) 56% (c) 43% (d) 31% b. 56% NSSE FY; Wisconsin = 55%

37. Striving for Deep Learning “Deep learning is learning that takes root in our apparatus of understanding, in the embedded meanings that define us and that we use to define the world.” J. Tagg (2003). The learning paradigm college (p. 70). Bolton, MA: Anker

38. Deep Learning Sub-ScaleIntegrative Learning • Worked on a paper or project that required integrating ideas or information from various sources • Included diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments • Put together ideas or concepts from different courses when completing assignments or during class discussions • Discussed ideas from your readings or classes with faculty members outside of class • Discussed ideas from your readings or classes with others outside of class (students, family members, co-workers, etc.)

39. Deep Learning Item:Integrative Learning Scale Students indicate: 1 = “never” to 4 = “very often”, they did the following during school year: [% = “very often” and “often”] • Worked on paper/project that required integrating ideas, info. from various sources • Included diverse perspectives (different races, religions, genders, political beliefs, etc.) in class discussions or writing assignments • Put together ideas or concepts from different courses when completing assignments or during class discussions • Discussed ideas from readings or classes with faculty members outside of class • Discussed ideas from readings or classes with others outside of class (students, family members, co-workers, etc.)

40. Compensatory Effect of Engagement - Student engagement positively related to FY and senior student grades and to persistence between the first and second year of college at the same institution - Engagement has compensatory effect on FY grades and persistence to the second year of college.

44. Using NSSE results in your explorations of first year student learning • Take a look at the instrument. What items or scales might you want to know how your first year students score? - what comparison group (to seniors, peers at other similar institutions, among departments, criterion reference) is compelling? • What items do you want to know more about what students responses really mean? • What does “prompt feedback” mean to students? • What assignments do students think require them to “synthesize ideas”? • How else might you assess the effective educational practices on NSSE? • How might you use NSSE results to warrant or add to an inquiry project of interest to you?

45. Approaches to using NSSE in inquiry about teaching and learning • Assess engagement in your class. • Add selected NSSE items to your course evaluation to obtain classroom level engagement data, compare to aggregate FY data • Use NSSE to pinpoint FY results deserving of additional inquiry • Ex: FY student faculty engagement low. Conduct student – faculty focus groups to discover obstacles to interaction • Examine group differences • Identify FY groups of interest (e.g., rural students, first-generation, majors, women in science) -examine difference • Assess impact of intervention • Ex: Effect of FY seminar on engagement; impact of service learning course on reflective learning and civic engagement outcomes

46. Localizing Findings • Variation in levels of student engagement within the university is greater than variation between universities • Improvement initiatives might best be designed and implemented at the college level (rather than based on institutional findings) to maximally impact overall student engagement on campus.

47. Localizing Findings cont’d • Even when institutional outcomes are disaggregated to the college level…less desirable engagement outcomes are attributed to shortcomings in another academic department • “These less-engaged students are not my students” • Many institutions have increased their institutional NSSE sample or done targeted, over-sampling efforts to specific majors.

48. Localizing Findings cont’d • Disaggregating NSSE outcomes to the department level often leads to the same series of observations that occurred at the college level • a faculty member within the department will often explain that“these less-than-optimal levels of engagement are not characteristic of students in my classes, but rather must be in other course offerings within the department. These are not my students!”

49. CLASSE (Classroom Survey of Student Engagement)created by Bob Smallwood, Univ. North Florida and Judy Ouimet, Univ. of Nevada, Reno • CLASSE is a pair of survey instruments that enable one to compare what engagement practices faculty particularly value and perceive important in a designated class with how frequently students report these practices occurring in that class. • CLASSEStudent is the survey instrument completed by each student enrolled in the designated class, while CLASSEFaculty is the survey instrument completed by the faculty instructor of the designated class.

50. Approaches to using NSSE in inquiry about teaching and learning • Assess engagement in your class. • Add selected NSSE items to your course evaluation to obtain classroom level engagement data, compare to aggregate FY data • Use NSSE to pinpoint FY results deserving of additional inquiry • Ex: FY student faculty engagement low. Conduct student – faculty focus groups to discover obstacles to interaction • Examine group differences • Identify FY groups of interest (e.g., rural students, first-generation, majors, women in science) -examine difference • Assess impact of intervention • Ex: Effect of FY seminar on engagement; impact of service learning course on reflective learning and civic engagement outcomes