Conceptualizing and Understanding Studies of Student Persistence

1. Conceptualizing and Understanding Studies of Student Persistence University Planning, Institutional Research, & Accountability April 19, 2007

2. Overview • Framing the persistence problem • Understanding results of retention studies • Providing perspective on concepts using IUPUI example

3. Framing the Problem • How should we define and measure persistence? • Graduation rates • An entering cohort approach • Probability of graduating within 150% of program length • How do these rates vary by student characteristics? • Time to graduation • A graduating cohort approach • Number of years (months) from matriculation to graduation • How does this time vary by student characteristics?

4. Framing the Problem • How should we define and measure persistence? • Retention/departure measured at a single interval • Between two academic years • Between two semesters • Within a single semester These three approaches assume time-invariant predictors: The effects of characteristics on retention/departure (or even the characteristics themselves) do not change over time

5. Framing the Problem • How should we define and measure persistence? • Retention/departure measured at multiple intervals • Can capture timing of departure • Assume time-variant predictors of retention/departure • Account for changes to the student body due to self selection However… Methods for examining persistence under this framework can be very complex and are relatively new to many in IR

6. Framing the Problem • How should we define and measure persistence? • Type of departure most often studied • Return vs. Do not return (in most general sense) • Other possible characterizations • Continuous Enrollment vs. Stop-out vs. Permanent absence • Transfer vs. Dropout (from higher education) • Voluntary withdrawal vs. Academic expulsion

7. Understanding Retention Results • Most common approach to study of persistence • Retention/departure measured at a single interval • Interval: Academic year • Dichotomous outcome: Return vs. Do not return • e.g., Second year retention among first-time students • Methods for dichotomous outcomes • More common: Logit (a.k.a. logistic regression) • Less common: Probit

8. Understanding Retention Results • Three common formats for presenting results • Used least often: Predicted probabilities • Used more often: Changes in probability (Delta-p) • Used most often: Odds ratios • All formats are related (and as such, are easily confused) • So what’s the difference?

9. Understanding Retention Results • Predicted probabilities • Two common approaches: • Ceteris paribus (i.e., all else being equal) • Isolates the “effect” of a particular characteristic (e.g., gender) • Assumes that students are average on all other characteristics • All else being equal, Females = 0.85, Males = 0.75 • Hypothetical (within reason!) student • Allow multiple characteristics to vary • Nonresident male with $2000 unmet need = 0.35 • Resident female with $0 unmet need = 0.90

10. Understanding Retention Results • Delta-p (i.e., change in probability) • Based on ceteris paribus approach • The female “effect” = female prob. – male prob. • 0.85 – 0.75 = 0.10 • Beware the misinterpretation of the delta-p! • Correct: A ten percentage point difference in prob. • Incorrect: A ten percent difference in prob. • What is the percent diff? (0.85 – 0.75)/ 0.75 = 13%

11. Understanding Retention Results • Odds • P/(1 - P) = Odds • Females: 0.85/(1 - 0.85) = 5.7 • Males: 0.75/(1 - 0.75) = 3.0 • Odds ratio (literally the ratio of two odds) • Odds ratio for females versus males 5.7/3.0 = 1.89 • Odds ratio for males versus females 3.0/5.7 = 0.53

12. Understanding Retention Results • Interpretation of odds ratios • OR ~ 1 = No difference in odds • OR > 1 = Greater odds (females have greater odds than males) • OR < 1 = Lower odds (males have lower odds than females) • OR can be expressed in terms of percentages • OR 1.89 = 89% greater odds • OR 2.89 = 189% greater odds • OR 0.53 = 47% lower odds

13. Understanding Retention Results • Beware the misinterpretation of odds ratios! • Compared to males: • Correct: Females have 89% greater odds... • Incorrect: Females have an 89% greater probability… • Incorrect: Females have an 89% greater likelihood…

14. Understanding Retention Results • Advantage of Delta-p • Discrete change in probability is more intuitive Remember: Delta-p is not equal to % change! • Limitation of Delta-p • Delta-p is assessed for the “average” student • “Average” student ~ overall probability • Logistic “probability” curve is not linear • Size of delta-p depends on overall probability • Practical significance not contextualized via overall probability

15. Understanding Retention Results • Limitation of Delta-p (continued) • Logistic Curve

16. Understanding Retention Results • Limitation of Delta-p (continued) • If overall probability were ~ 0.50

17. Understanding Retention Results • Limitation of Delta-p (continued) • If overall probability were ~ 0.80

18. Understanding Retention Results • Advantage of Odds Ratio • Is not tied to location within distribution

19. Understanding Retention Results • Limitations of Odds Ratio • What’s an odds ratio again? (Not intuitive) • Is not tied to location within distribution! • Female odds are 3 times greater than odds for males! • Sounds like a big deal. Is it? It depends… • Overall prob 0.50, Delta-p = 0.27 Wow! • Overall prob 0.98, Delta-p = 0.03 Hmph!

20. Understanding Retention Results • Predicted Probabilities: Why I like ‘em… • Most intuitive approach to presenting results • Can be calculated ceteris paribus or hypothetical • Can easily derive Delta-p from probabilities • Final Precaution • Any of these formats for presenting results are only as good (i.e., accurate or plausible) as the statistical model from which they are derived

21. Understanding Retention Results • Questions to ask yourself (or others!) • How are the results reported? • Predicted prob., delta-p, or odds ratios • If reported as odds ratios… • Are odds ratios being correctly interpreted? • i.e., reported as % change in odds? • If reported as delta-ps… • Are delta-ps being correctly interpreted? • i.e., reported as percentage point change in probability? • To assess practical sig. of delta-p, is overall probability provided?

22. Perspective: An IUPUI Example • A different look at IUPUI’s one-year retention rate • Considers one-year retention rate as set of sequential decisions • Retention between fall and spring semesters • Retention between spring and second academic year • Two outcomes, two models • Different than single model: beginning of first to second year • Assumes reasons for retention/departure differ over time • Uses time-varying predictors to capture differential “effects” • Sample: IUPUI full-time beginners (2004 and 2005 cohorts)

23. Perspective: An IUPUI Example Full-time Beginner Cohort Spring Did not Return Returned 14% 86% Fall Did not Return Returned 26% 74%

24. Perspective: An IUPUI Example • Predictors of retention (time invariant) • Age (20+ vs. less than 20) • Gender (Female vs. male) • Race (Hispanic, African American vs. other race) • State residency (Non-resident vs. resident) • Campus residence (Live on campus vs. live off campus)

25. Perspective: An IUPUI Example • Predictors of Retention (time variant) • Credit load earned (Full-time vs. less than full-time) • Semester GPA • Completed FAFSA • Second semester = Current year • Second year = Reapplied for subsequent year • Unmet need (i.e., need – total aid) • Net aid (i.e., total aid above need)

26. Perspective: An IUPUI Example • Significant Predictors of Second Semester Retention (Remember: “All else being equal”) • Race • Hispanic prob. = 0.91, • Other race prob. = 0.85 (not including African Americans) • Fall credit load earned • Full-time prob. = 0.89 • Part-time prob. = 0.80 • FAFSA for current year • Completed prob. = 0.87 • Did not complete prob. = 0.76

27. Perspective: An IUPUI Example • Significant Predictors of Second Semester Retention (Remember: “All else being equal”) • Fall Semester GPA Probability

28. Perspective: An IUPUI Example • Significant Predictors of Second Semester Retention (Remember: “All else being equal” except FAFSA and Net Aid) • Unmet Need Probability

29. Perspective: An IUPUI Example • Significant Predictors of Second Year Retention (Remember: “All else being equal”) • Age • 20+ prob. = 0.66 • < 20 prob. = 0.75 • Campus residence • On campus prob. = 0.69 • Off campus prob. = 0.75

30. Perspective: An IUPUI Example • Significant Predictors of Second Year Retention (Remember: “All else being equal”) • Spring credit load earned • Full-time prob. = 0.78 • Part-time prob. = 0.67 • FAFSA for subsequent year • Did not reapply prob. = 0.51 • Reapplied = 0.77 • Newly applied = 0.92

31. Perspective: An IUPUI Example • Significant Predictors of Second Year Retention (Remember: “All else being equal”) • Spring Semester GPA Probability

32. Perspective: An IUPUI Example • Significant Predictors of Second Year Retention (Remember: “All else being equal” except FAFSA) • Subsequent year unmet need and net aid Probability

33. Perspective: An IUPUI Example • Summary • Time invariant predictors get “turned on” at different times • Second semester: Race • Second year: Age, Campus residence • Time variant predictors have differential “effects” • Unmet need isn’t as strong a predictor of second year retention • May be due to self selection after first semester • May be due to a failure to reapply “effect”

34. Conceptualizing and Understanding Studies of Student Persistence University Planning, Institutional Research, & Accountability April 19, 2007

35. Other Pertinent Issues • Financial Aid Effects and False Attribution • Type and amount of aid awarded is tied to criteria (student characteristics) that also predict retention • Example 1 • Lower income  lower prob. of persisting • Lower income  more need based aid • More need based aid  lower prob. of persisting • Example 2 • Higher SAT  higher prob. of persisting • Higher SAT  more merit aid • More merit aid  higher prob. of persisting

36. Other Pertinent Issues • Financial Aid Effects and False Attribution • Research results have been inconsistent as a result • Must do more to separate effects of selection criteria from effects of dollar amount • IR and other higher education research just starting to touch on this issue