The Synergy of and Readiness for High Impact Practices During the First-Year of College

1. The Synergy of and Readiness for High Impact Practices During the First-Year of College Michele J Hansen, Ph.D. Executive Director of Research, Planning, & Evaluation mjhansen@iupui.edu Lauren Chism, M.S., Director of Themed Learning Communities lchism@iupui.edu Daniel J. Trujillo, M.S. • Qualitative Research Analyst djtrujil@iupui.edu University College

2. Presentation Overview • Literature Review • Research Setting & Interventions • Research Questions • Research Methods • Results • Discussion • Limitations and Guidelines for Future Investigations • Questions! • Presentation found at: http://research.uc.iupui.edu/

3. Literature Review • Summer Bridge Programs • First Year Seminars • Learning Communities

4. Summer Bridge Programs • Summer programs designed to assist students’ transition to college • Generally include attention to study skills, writing and math. • Positive outcomes have been associated with program participation • More research needed

5. First Year Seminars • Common intervention for beginning students •  Program Objectives (Padget & Keup, 2012) • Developing academic skills • Developing a connection with the institution • Providing an orientation to campus resources & services • First Year Seminars are often linked with increases in retention (Pascarella & Terenzini, 2005) • Mixed results in the literature regarding the impact on GPA • Higher GPA (Friedman & Marsh, 2008) (House, Rode & Xiao, 2008) (Schwartz & Grieve, 2008) • Higher GPA only in select groups (Dalgreen, 2008) (Hansen, Williams & Chism, 2008) • No impact on GPA (Rogers, Eglsaer, Muehsam, Cailouet & Kan, 2008)

6. Learning Communities • Cohort of students enrolling in linked courses, typically connected through an interdisciplinary theme

7. Learning Communities • Most institutions have learning communities programs, but not all students are able to participate: • 62% reported having learning communities (Barefoot, 2002) • 18% of NSSE participants reported participating in learning communities. (NSSE, 2011) • Increases & retention & GPA • Learning communities are designed to engage students in learning, retention gains are not the purpose, but a welcome byproduct. (Lardner & Malnarich, 2008)

8. Conducting Research with Students Participating in Multiple Interventions • Studies often lack a control group (Kezar, 2000) • Random assignment is rare. • No gains in persistence for 16 randomly assigned learning non-traditional learning community students at a proprietary technical institution. • (Goldberg & Finkelstein, 2002) • Gains in retention for first year seminars (Strumpf & Hunt,1993) • Students participate concurrently in multiple interventions • While most learning community programs include a first-year seminar, participation in first-year seminars is not accounted for in many studies examining the impact of learning communities on grade point averages (Andrade, 2007-8). • Huber (2010) has completed some preliminary research looking at the effects of multiple high impact practices on graduation rates based on National Survey of Student Engagement (NSSE) data.

9. Research Setting • Ranked 3rd "up and coming" American universities and Recognized for Learning Communities & the First Year Experience (U.S. News and World Report) • 8th best public college in the Midwest (Forbes) • Student population of just over 30,000 students • Over 250 degree programs from both Indiana & Purdue Universities, guided by the Principles of Undergraduate Learning • University College serves as entering unit. • Majority of FT, FT students commute to campus (73%) • Many FT, FT are first generation college students (41%) • Many FT, FT receive a Federal Pell Grant (44%)

10. Intervention Descriptions • Common elements of 3 interventions • (Summer Bridge, First Year Seminars and Themed Learning Communities) • Designed for entering freshmen • Cohorts of 25 students • Instructional team • Faculty member • Academic Advisor • Student Mentor • Librarian

11. Summer Bridge Program Essential elements: • 2 week program prior to fall classes introduced to collegiate-level expectations for writing, mathematics, communications, critical thinking • become more acquainted with the campus, and learn important study skills. • More information available at: http://bridge.uc.iupui.edu/

12. Summer Bridge: Providing a Sense of Readiness • 421 First-Year students participated in 2010 (240 subsequently participated in a TLC). • Completed an anonymous end-of program questionnaire. • Included both Quantitative and Qualitative Items. • Allowed for students’ opinions and perceptions of the program to be gathered.

13. Campus Navigation & Locating Resources Note: Responses provided on a Likert scale ranging from1 = Strongly Disagree to 5 = Strongly Agree

14. Diversity Awareness & Appreciation Notes: Responses provided on a Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree.

15. Academic Success: Critical Thinking, Writing, Math, & Public Speaking Notes: Responses provided on a Likert scale ranging from 1 = Strongly Disagree to 5 = Strongly Agree

16. Examples of Actual Student Comments • “I think learning about all the resources available to students”. • “Learning college etiquette, exploring diversity, study skills”. • “Lives up to its name, bridge between high school and college”. • “Helped get me into the college mindset and show me college level skills”. • “Expectations of how college was going to be liked were changed”. • “Learning what to expect in college and how to be an organized student”. .

17. First Year Seminars Essential elements: • 1-3 credit course • Introduces academic skills, key campus resources, and creates opportunities to connect with faculty and staff.

18. Themed Learning Communities (TLCs at IUPUI) Essential elements: • 3 or more linked courses • Interdisciplinary theme & connections • Learning beyond the classroom • More information available at: http://tlc.iupui.edu

19. C. • TLC • First Year Seminar Students Participating in Multiple Interventions and High Impact Practices • Summer Bridge A. B. D. 2010 Students Participating: A=1018, B= 142, C= 400, D=240

20. Research Questions • 1. Is participation in high impact practices and a summer bridge intervention significantly positively related to higher levels of academic success (first-year GPA and one-year persistence rates), even when considering student characteristics? • 2. Does the synergy of multiple high impact practices (themed learning community with an embedded fist-year seminar) contribute to students’ academic success levels more than high impact practices in isolation or no participation, even when considering student characteristics?

21. Research Questions (continued) • 3. Does participation in a summer bridge program prior to participation in multiple high impact practices interventions (a TLC with an embedded first-year seminar) contribute to academic success levels more than participation in high impact interventions without Summer Bridge, even when considering student characteristics? • 4. Is participation in a Summer Bridge-TLC significantly and positively related to higher first year grade point averages after accounting for the confounding effects of self-selection?

22. Research Methods and Data Analyses • Quasi-experimental design • Hierarchical Linear Multiple Regression • Hierarchical Logistic Multiple Regression • Instrumental Variable Analysis in an attempt to account for self-selection • Participants: 2028 FT, FT regularly admitted students in 2010 cohort

23. Random Assignment Not Feasible • Many students would have been denied access to programs (ethics). • Logistical issues with registration and making sure that only students in the “experimental group” could enroll in the sections (administrative). • Political pressures coming from legislators that low-income, first-generation students should participate in academic support interventions to help them make successful transitions to college (political).

24. Variables (Multiple Regressions) • Outcome Variables: • First-Year GPA • One-Year Retention Rate • Interventions (entered in Step 2; dummy coded) • First Year Seminar (no TLC or Summer Bridge) • Themed Learning Community-FYS (no Summer Bridge) • Summer Bridge (no TLC-FYS) • Summer Bridge – TLC-FYS Covariates (entered in Step 1): • Academic Preparation (HS GPA, SAT scores) • Gender (Female 1, Male 0) • Low Income (Pell Grant 1, No Pell Grant = 0) • Proxy for Student Motivation (weeks since admit date)

25. Why IV Analysis? • Causal inference challenging in quasi-experimental designs - possibility of hidden bias. • Hidden bias may exist as a result of failure to control for unobservable factors. • Threats to internal validity such as self-selection. • PSM not used because large samples are required, hidden bias may remain because matching only controls for observed variables (to the extent that they are perfectly measured) (Shadish, Cook, & Campbell, 2002).

26. Instrumental Variable Analysis • A variable z is called an instrument or instrumental variable for the regressorx in the regression model y = x+uif (1) z is uncorrelated with the error u; and (2) z is correlated with the regressorx. • Can solve the problem of omitted variable bias – IV can help solve the problem of missing or unknown variables when random assignment not possible.

27. Fundamental IV Assumptions • The instrument must be correlated with the endogenous explanatory variables (Summer Bridge –TLC Intervention). • The instrument cannot be correlated with the error term in the explanatory equation. The instrument cannot suffer from the same problem as the original predicting variable (test of overidentifyingrestrictions, such as Sargan test).

28. IV Five Assumptions • Stable Unit Treatment Value Assumption (SUTVA) – “spillover effects” • Random assignment • Exclusion restriction • Nonzero average causal effect of instrument on treatment • Monotonicity (see Angrinst & Pischke, 2009 and Porter, 2012)

29. Bane of IV Analyses • Weak instruments may produce inconsistent and inefficient results. • Valid and reliable instruments very difficult to locate.

30. Examples of Instruments Used

31. Instruments Considered • Campus Housing • Distance from Campus (based on zipcode) • HS Class Size • Applied for Financial Aid • Twenty-First Scholar Participant

32. Current Study Instruments • Confidence in Timely Degree Completion • Placement in English Composition I

33. Evaluating the Instruments • Met two fundamental assumptions. • Strongly correlated with SB-TLC • Robust F statistic was 15.28 (df = 7, 2020; p < 0.001). This F value was above the threshold of 10.00 recommended by Stock and Watson (2007). • The results of Sargan and Basmann tests of overidentifying restrictions were not statistically significant, indicating that the model was appropriately specified and the instruments were not correlated with the error term.

34. Word On Instruments • “All instruments arrive on the scene with a dark cloud of invalidity hanging overhead. This cloud never goes entirely away, but researchers should chase away as much of the cloud as they can.” • Murray, 2010

35. Hierarchical Multiple Regression: Cumulative First Year GPA (N=2028)

36. Results Hierarchical Multiple Regression: One-Year Retention (N=2028)

37. Two Stage Least Squares Results for the IV Model

38. Two Stage Least Squares Results for the IV Model

39. High Impact Practices “When I am asked, what one thing we can do to enhance student engagement and increase student success? I now have an answer: make it possible for every student to participate in at least two high impact activities during his or her undergraduate program, one in the first year, and one taken later in relation to the major field. The obvious choices for incoming students are first-year seminars, learning communities, and service learning” (George Kuh, 2008)

40. First-Year Seminar • No positive effect for FY GPA once background characteristics were accounted for and considered in conjunction with the other interventions in the hierarchal linear multiple regression model. • Positive effect on one-year persistence. • Vital component of TLC and SB • Instructional team model • Designed help students adjust, make connections, career development and exploration.

41. Themed Learning Community and Synergy of High Impact Practices • TLC intervention made a more substantial contribution to GPAs than the Summer Bridge intervention or the standalone FYS. • Synergy of multiple high impact practices (TLC with an embedded FYS) contribute to students’ academic success levels more than high impact practices in isolation or no participation, even when considering student characteristics.

42. Summer Bridge • Summer Bridge intervention significantly positively predicted retention rates and cumulative first-year GPAs. • May be an ideal early intervention strategy for counterbalancing any deficits that students have resulting in a lack of readiness to begin college. • Broad-based intervention not restricted to particular groups such as low-come students from underrepresented minority groups.

43. Providing a Readiness for High Impact Practices: Summer Bridge-TLC • Summer Bridge-TLC contributed to academic success levels more than other interventions. • Possible that the Summer Bridge intervention helped prepare students and provide a sense of readiness for the engaging and powerful pedagogies. • Summer Bridge intervention may have provided the foundation of support and community that allowed students to fully reap the benefits offered in their TLC-FYS experiences.

44. Limitations • Generalizability • Only Two Short Term Outcomes (FY GPA and One-Year Persistence) • Results indicate if effective, but not why - what components of TLCs (service learning, interdisciplinary theme, integrated assignments, sense of belonging and community) produced positive outcomes. • Limitations of IV analysis prevent ability to make casual inferences.

45. Limitations in Post-Secondary Educational Contexts • Treatment difficult to standardize. • Faculty own curriculum. • TLC program really consists of multiple interventions as faculty teams collaborate on identifying unique interdisciplinary theme for their section. • Meeting students diverse needs and notion of “one-size does not fit all.”

46. Future Investigations • Multi-institutional studies. • Investigate more outcomes such as learning, graduation, and degree attainment. • Mixed-Methods (qualitative investigations of program implementation strategies to supplement quantitative methods). • Variation within program studies. • Consider random assignment for pilots (new untested program that would be small enough not to deny tx to large groups of students). • Consider RA to different types of tx rather than not providing services/programs or waitlist controls.

47. Questions!