Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates

1. Estimating the Causal Impact of Financial Aid Programs on Student Retention Rates A Non-Technical Introduction to RD and ITS Designs Michigan Association for Institutional Research Nov. 6-8, 2013 Grand Rapids, MI Reuben Ternes Oakland University OIRA ternes@oakland.edu

2. Increased Scrutiny Over Retention Rates at OU • Rates now tied to funding • Historical retention rates between 70% and 75% • President requested a review of the impact that our financial aid policies had on retention rates. • This presentation details the results and methods used to conduct that review. Intro: Retention

3. Intro: Retention • Section I: Policy Overview • Section II: Regression Discontinuity Designs • Section III: Interrupted Time Series Designs Presentation Outline

4. OU aid is very transparent • Published criteria given to public • This presentation covers need-based aid only • Two different types of need aid at OU: • 1) Academic/Housing grant – Must have a HSGPA of a 3.0 or higher and an ACT of 21 or more. Must demonstrate need. Up to $8k. • 2) 100% Tuition Award – Covers gap between tuition and EFC after all other aid is calculated. I – Policy Overview

5. Academic/Housing Grant: Contains very hard criteria for inclusion. Suitable for RD analysis. • 100% Tuition Award: Sudden onset. Suitable for ITS analysis. I – Policy Overview

6. RDD = Regression Discontinuity Design • How can you assess program effectiveness if you separate participants based on expected performance? • Specifically: How can we be sure that our financial aid program improves retention rates, when the aid that we give is confounded with prior student ability? • (ACT scores & HS GPAs are positively correlated with retention rates) II – RD Overview

7. Basic Logic – When you are looking at an outcome and relating that outcome to some measure, but have a sudden change of group status, a discontinuity will be visible at that change if group membership has an impact on your outcome. II – RD Basic Logic

8. II – RDD Visual Example

9. People with very similar ‘assessment’ scores are similar. • If group membership is controlled by a strict cut-off • and similar people have very different experiences • then a gap should be visible if the group membership impacts the outcome • The above logic is extremely useful and powerful • The mathematics of RD is about establishing unbiased treatment effects similar to random assignment. II – RD Logic

10. The logic of RD is simple & intuitive • Often times, visuals are enough to confirm causation and guide policy decisions • (i.e. no regression is actually required) • Visuals are very convincing to many people • Unlike standard regression techniques, RD’s logic is inherently causative, not correlative. II – RD Visuals

11. Do our grants cause student to stay at the university? • If so, can we guess how many students it impacts? II – RD in Action

12. First, establish that nothing strange is happening at cut-off for students that do not receive the grant. • (i.e. control condition) • Technically, you don’t need this step, but it does help convince some people. • (What you are about to look at is a graph of retention rates by ACT score. Each ACT score has its own retention rate, which is the collective rate for each student with that score. Each ACT score is composed of hundreds of students). II – RD in Action, Step 1

14. Now examine what happens at the cut-off for students that do receive the grant. • Notice the large, obvious gap! • This discontinuity is where the method gets its name from. II – RD in Action, Step 2

16. Now graph them together, so people can see them in relation to each other. • Ignore the behavior near the extreme tails. It has no real meaning within an RD framework. • Also, the N is smaller for extreme ACT scores. RD in Action, Step 3

18. The evidence in this case is very clear. • Our need-based aid improved retention rates • Estimating by how much is harder. • For the students near the cut-off, retention rates improved by about 10%. • Based on this data, we reduced the criteria to obtain the grant, so more students would qualify. • We will continue to monitor the progress of these students now that the criteria has changed. • No theoretical reason for the impact to be equivalent across all potential ACT scores. II – Policy Changes

19. RD is not better than random assignment • It can only produce estimates near the cut-off. • The effect size in the tails is unknown • Obtaining unbiased treatment estimates is still a challenge • Must model the functional relationship exactly – linear, quadratic, cubic, something else? • ‘Fuzzy’ RDs, where the cut-off criteria is mostly adhered to, present additional challenges II – RD Caveats

20. ITS = Interrupted Time Series Design • Logic: Essentially it is an extension of ‘before’ and ‘after’. III - ITS

21. Control Treatment Control Treatment How ITS Shows Causation

22. Subjecting the same group to different conditions, repeatedly, creates a very strong case for causation. • If differences are found, causation is only threatened if something else varies alongside the timing of the treatment/control condition. • In most cases, it is very unlikely anything else will vary in such a manner. ITS & Causation

23. Opportunities with an ideal ITS in educational research are very rare. • We rarely can subject the same sample to repeated conditions. • Mostly, we get ‘before and after’ conditions, often with different populations (i.e. years) ITS in Educational Research

24. But the basic principles of ITS are still valuable! • In many cases a large sample size mitigates most potential issues of not using the same sample. • Though repeated measurement is best, a single measurement can still be persuasive. ITS – Still Valuable!

25. Remedial Math class. ITS – Example 1

26. In 2005, Institution X created Remedial Math 3 • Designed to address deficiencies with some students in Remedial Math 2. • New placement scores were developed for several courses. • Because this policy change happened suddenly, it is possible to analyze data from this time period as a ‘natural experiment,’ comparing students from before the policy change (2004) and those after the policy change (2005). Background

27. Percentage of Students that Complete the Entire Remedial Math Sequence Two Year Completion Rate Math 3 Implemented Here These Students Take Math 2 These Students Take Math 3 There was a significant drop in 2-year completion rates for Math 1 just after the implementation of the new Remedial Math 3 course. The sample size for each year is close to 1,000

28. Retention rates & the FYAC ITS – Example 2

29. ITS – Example 2

30. Highest Year-to-Year Variance (pre-2012): 3.1% • Average Year-to-Year Variance (pre-2012): 1.5% • 2011-2012 Variance: 7.8% • That is more than double the previous high, and five times more than average! • Q: What Happened? • A: We changed our entire approach to first year advising and created a first year advising center (FYAC) • Classic example of potential ITS. • Strong and sudden onset of a novel environment ITS – Example 2

31. There are more internal threats to validity with ITS than there are with RD. • In using this technique, you will still probably have to think about plausible alternatives and see if you rule them out. • Two plausible explanations still exist in this example: • 1) Regression to the mean (2011 had a very low retention rate, so year-to-year variation could be artificially high). • 2) The 2012 cohort also had the highest incoming academic profile. • Could this combination explain the results? ITS – Example 2

32. Simple regression estimates suggest that some of the variation is explained by the alternative explanations. • About 3% is still left unexplained though, which bodes well for our FYAC (and our decision to create it). • The explanation can be graphed visually! • Graph displays 2012 cohort, 2011 cohort, and historical rates for comparison. • The tails of the distribution have low N. Generalize here only with great caution. ITS – Example 2

33. Retention Rates by ACT Scores

34. Two different types of need aid at OU: • 1) Academic/Housing grant – Must have a HSGPA of a 3.0 or higher and an ACT of 21 or more. Must demonstrate need. Up to $8k. • 2) 100% Tuition Award – Covers gap between tuition and EFC after all other aid is calculated. • We will now focus on #2, the 100% tuition award. • Visually, you do not have to present ITS as a timeline – there are other ways to think of it, such as comparing historical rates to current rates. ITS – Example 3

35. What impact did our 100% tuition award have? • Analyze data by Expected Family Contributions • In theory, students with high EFCs won’t get much award money, and probably don’t need as much. • So impact should be different across EFC groups. • Plot retention rates by EFC for historical rates, and then under the 100% tuition award. ITS – Example 3

36. Retention Rates by EFC

37. Possible Interpretations • The results were somewhat unexpected, as very low EFC students did not benefit from the aid. • Possibly the aid just wasn’t enough to impact their retention decision, because the amount of money just won’t cover living expenses. • As EFC rises, the aid, though less on average, becomes more of a decision factor for these families. • Threats to Validity • Only one control year. • The economic recession is difficult to model, could cause a great deal of variance in unexpected ways. ITS – Example 3

38. In general, ITS is easier to use than RD. • But it generally has more threats to validity. • It’s a great tool to try and estimate causality, but it is far from perfect. ITS - Conclusions

39. Thank you! The End