What Works in Promoting Student Success

1. What Works in Promoting Student Success By Steve Robbins AVP Research, ACT, Inc. Presented as part of the University of Michigan Forum on Diversity, Merit and Higher Education

2. Putting the Pieces Together & Hoping They Work Concept and original artwork by Jorge O. Calvillo, Canyon Springs High Schools, Moreno Valley, California, winner of the ACT High School Impact poster contest.

3. Agenda Why College Students Stay What Works in Promoting College Success Risk, Service Use, & Success: A Case Study Putting It All Together

4. About ACT Research: The “Propeller Heads” • Access to Longitudinal Data • 8th - 10th – 12th linked files 140,000 per cohort • ACT/COMPASS linked to college transcript/outcomes (n = 1 million and counting) • Student Readiness study: 15,000 students at 48 2- and 4-year institutions moving into year 6 • Use of National Student Clearinghouse data • Policy, Statistical, Measurement, Career Transition, I/O, and Survey Research Staff

5. Harris, S. (1991) Can’t you guys read? Cartoons on Academia. Rutgers University Press. New Brunswick, NJ, p.74

6. Why College Students Stay: 4-year Colleges • First-year GPAhas large effects on likelihood of retention and transfer; • Motivation (Academic Discipline) and pre-collegiate academic preparationhave indirect effects on retention and transfer by working through 1st-year GPA • Social connectionhas a direct effect on retention. • SESpredictive of transfer behavior: • Higher SES students transfer while poor students give up • African-American Students have high commitment but difficulty with classes resulting in higher drop-out rates. Robbins et al. (2006) Allen et al. (in press)

7. Why College Students Stay: 2-year Colleges • Pre-collegiate academic preparationis the strongest predictor of all outcomes; • Motivation(Academic Discipline) distinguishes retained and graduating students from transfer and drop out • Social connectionhas effects only for those students who transferred to 4-year institutions; • Socioeconomic statusdistinguishes all groups from drop-out: higher SES kids are likely to transfer and low SES kids drop out

8. Common Findings across 2- & 4-year Studies • Academic preparation, Socio-Economic Status (SES) and Academic Discipline are all critical • 1st year GPA essential for 4-year students • Students socially connected are more likely to transfer upon 2 year graduation or stay (4 year)

9. “We’re looking for a more comprehensive research strategy than simply ‘Google it.’” Harvard Business Review. (July-August 2007)

10. What Works . . . • Summarizing the effect of something over multiple data points • Create confidence intervals of the true effect size • Interpretation of multiple studies, better than any individual study Meta-Analysis and Validity Generalization as Key Tools

11. Testing Integrated Meta-Analytic Path Analysis Motivation/Skills Performance & Persistence InterventionTypes Self-Regulation Social Engagement The Effects of College Interventions on College Outcomes as Mediated by PSF’s Robbins et al. (2007)

12. Categorizing College Interventions • Orientation(21 hours) – summer, early fall, time-limited • Freshman Year Experience(45 hours) • Academic (8 hours) • Study skills • Learning strategies • Note-taking • Self-Management (6 hours) • Stress management • Self-control • Anxiety management • Hybrid of Academic & Self-Management (12 hours)

13. Categorizing Psychosocial Factors (PSFs) 3 Categories: • Motivation: • Academic Discipline • Commitment to College • Self-Regulation: • Emotional Control • Academic Self-Confidence • Social Engagement: • Social Connection • Social Activity Robbins, S., Allen, J., Casillas, A., Peterson, C., & Le, H. (2006) Robbins, et al. (2004)

14. Effects of Intervention on Outcomes EFFECT SIZE: 0 = No Effect .1 - .2 = Small .2 - .4 = Moderate .4+ = Strong

15. Effects of Intervention on Psychosocial Factors EFFECT SIZE: 0 = No Effect .1 - .2 = Small .2 - .4 = Moderate .4+ = Strong

16. Effect Sizes of PSFs on College Outcomes EFFECT SIZE: 0 = No Effect .1 - .2 = Small .2 - .4 = Moderate .4+ = Strong Robbins, et al. (2004)

17. Testing Integrated Meta-Analytic Path Analysis PSFs - Motivation - Self-Regulation Indirect .11, .11, -.13 Total .24, .21, .57 • Intervention • Academic • Self-Mgt • Hybrid GPA Direct .13, .10, .70 About 50% of the effect of Academic and Self-Mgt interventions on GPA are through relevant PSFs The effect of Hybrid intervention on GPA is fully direct

18. Testing Integrated Meta-Analytic Path Analysis • PSFs • - Motivation • Self-Regulation (only for Self-Mgt) • Social Engagement (only for FYE) Indirect .06, .05 Total .27, .05 • Intervention • Self-Mgt • FYE Persistence Direct .21, .00 Self-Mgt Intervention has a strong direct effect on persistence Though small, 100% of the effect of FYE intervention on persistence is through relevant PSFs

19. What it means: • Interventions with academic focus are key • Boost academic interventions using self management strategies, i.e., Hybrid • Align specific interventions to narrowed outcomes (PSF and/or success) to increase treatment effect • Rethink goals & focus of Freshman Year Experience • Understand mediating role of motivation and self-regulation factors to promote student success

20. Giving Guidance Herzog & Miller (1985)

21. Risk, Service Use, & Success: A Case Study • Public Southwestern University • 4-year Institution • Over 13,000 Undergraduate Students and 1,100 Faculty • 31% Dropout Rate after Freshman Year • Implemented Card Swiping System to Monitor Resource Use Robbins et al. 2007

22. 3 Critical Elements • Coordinate resource & service use options • Academic services • Recreational resources • Social resources • Academic referrals • Advisory / career services • Determine student levels of risk by using admission test scores and Psychosocial Factors • Target and contact students for help

23. Program Model • Identify At-risk Studentsby testing entire entering class and identify bottom 24% on retention risk • Use SRI Scale scoresto match student needs with University and ACT/SRI resources (e.g., advising, FYE, academic support) • Provide interpretive feedback to students on importance of motivation and academic success • Evaluate results

24. Resource & Services Utilization: Robbins et al. (2007)

25. Association of Risk Level & Academic Service Use on Retention & 1st-year GPA .08 .24

26. Retention Recommendations based on Case Example • Designate a visible individual to coordinate a campus-wide Retention Planning Team • Conduct Systematic Analysis • Academic/Non-academic Characteristics/Needs • Persistors & Non-persistors broken down by race/ethnicity • Implement Early-Alert Assessment and Monitoring System • Academic/Non-academic Factors • Identify At-Risk Students

27. Putting It All Together Academic Factors (HS GPA + ACT) 68% Other Academic Psychosocial Discipline Factors 21% 11% • Psychosocial Factors Supplement but Do Not Replace Traditional Indicators

28. Ensuring Student Success • Academic preparation & performance are at the hub of all else • Be clear on Goals • Satisfaction, learning, & persistence are not the same • Be strategic in your use of resources • Move the Mountain to the students • Don’t be afraid of intrusive advising

29. Herzog & Miller (1985)

30. Questions?

31. References ACT, Inc. (2008). What We Know about College Success: Using ACT Data to Inform Educational Issues. Iowa City, IA: Authors. ACT, Inc. (2007). State of College Readiness for Latino Students. Iowa City, IA: Authors. ACT, Inc. & The National Council for Community and Educational Partnerships. (2007). Using EXPLORE® and PLAN® data to evaluate GEAR UP programs. Iowa City, IA: Authors. ACT, Inc. (2004). Schools Involving Parents in Early Postsecondary Planning. Iowa City, IA: Authors. ACT, Inc. (2002). Creating Seamless Educational Transitions for Urban African American and Hispanic Students. Iowa City, IA: Authors. Allen, J., Robbins, S., Casillas,A., & Oh, I.-S. (in press).Why college students stay: Using academic performance, motivation, and social engagement constructs to predict third-year college retention and transfer. Research in Higher Education Braxton, J., Sullivan, A., & Johnson, R. (1997). Appraising Tinto’s theory of college student departure. In J. C. Smart (Ed.) Higher Education: Handbook of Theory and Research, 12, 107-158. New York: Agathon. Bucheri, C., Hampton, T., & Voelker, V. (eds.) (1991). The Student Body: Great Cartoons from the Kappan. Phi Beta Kappa. Bloomington, IN.

32. References (cont.) DesJardins, S. L., Kim, D. O, & Rzonca, C. S. (2002-2003). A nested analysis of factors affecting bachelor’s degree completion. Journal College Student Retention, 4, 407-435. Habley, W. & McClanahan, R. (2004).What Works in Student Retention – All Survey Colleges. ACT, Inc. Iowa City, IA. Harris, S. (1991) Can’t you guys read? Cartoons on Academia. Rutgers University Press. New Brunswick, NJ Herzog, K. & Miller, M. P. (eds.) (1985).Scholarship: More Great Cartoons from the Kappan. Phi Beta Kappa. Bloomington, IN. Horn, L. & Nevill, S (2006). Profile of undergraduates in U.S. postsecondary education institutions: 2003-2004: With a special analysis of community college students (NCES 2006-184). U.S. Dept. of Education. Washington, DC: National Center for Education Statistics. Larson, Gary (1995). The Far Side Gallery 5. Universal Press Syndicate. Kansas City, MO Le, H., Casillas, A., Robbins, S., & Langley, R. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the Student Readiness Inventory. Educational and Psychological Measurement, 65, 482-508. Lotkowski, V., Robbins, S., & Noeth, R. (2004).The role of academic and non-academic factors in improving college retention. ACT Policy Report. Iowa City, IA: ACT, Inc.

33. References (cont.) Pascarella, E. T., & Terenzini, P. T. (2005).How College Affects Students: A Third Decade of Research. San Francisco: Jossey-Bass. Peterson, C. H., Casillas, A., & Robbins, S. B. (2006). The Student Readiness Inventory and the Big Five: Examining social desirability and college academic performance. Personality and Individual Difference, 41, 663-673. Porter, S.R. (2003-2004). Understanding Retention Outcomes: Using Multiple Data Sources to Distinguish Between Dropouts, Stopouts, and Transfer-Outs. Journal of College Student Retention: Research, Theory & Practice, 5(1), 53-70. Robbins, S. B., Allen, J. Casillas, A., Akamigbo, A., Saltonstall, M., Cole, R., Mahoney, E. & Gore, P.A. (2007). Associations of Resource and Service Utilization, Risk Level, and College Outcomes. Manuscript submitted for publication. Robbins, S. Allen, J., Casillas, A., Peterson, C., & Le, H. (2006). Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes. Journal of Educational Psychology, 98, 598-616. Robbins, S. B., Lauver, K., Le, H., David, D., Langley, R., & Carlstrom, A. (2004). Do psychosocial and study skill factors predict college outcomes? A meta-analysis. Psychological Bulletin, 130, 261-288. Robbins, S., Oh, I., Button, C., & Le, H. (2007). The effects of college interventions on psychosocial mediators and academic and persistence outcomes: An integrated meta-analysis. Manuscript submitted for publication. Swail, W. S. (2004, January 23). Legislation to improve graduation rates could have the opposite effect. The Chronicle of Higher Education, B16.

34. What Works in Promoting Student Success Correspondence regarding this presentation should be addressed to: Steve Robbins, AVP Applied Research, ACT, Inc., PO Box 168, Iowa City, IA 52243-0168 or email: steve.robbins@act.org phone: 319-337-1227