1 / 36

Marc Frenette

Preliminary : not for quotation. ***LILE and Aspirations question Reducing Barriers to Postsecondary Education Through Innovative Interventions: Evidence from Canadian Field Experiments McGill University Social Statistics Speaker Series March 21, 2012. Marc Frenette.

eydie
Download Presentation

Marc Frenette

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Preliminary: not for quotation ***LILE and Aspirations questionReducing Barriers to Postsecondary Education Through Innovative Interventions: Evidence from Canadian Field ExperimentsMcGill University Social Statistics Speaker SeriesMarch 21, 2012 Marc Frenette

  2. ReducingBarriers to Postsecondary: The Need for Hard Evidence • Improving access to PS important for many reasons: • Private gains (Card, 1999; many others..) • Societal gains, including productivity enhancement (Coulombe and Tremblay, 2006), crime reduction (Lochner and Moretti, 2004), longer life expectancy (Lleras-Muney, 2005), improved health (Arendt, 2008), and civic participation (Milligan, Moretti, and Oreopoulos, 2004) • Equalizing opportunities / increasing intergenerational mobility

  3. ReducingBarriers to Postsecondary: The Need for Hard Evidence • The Canadian literature has successfully identified several important barriers to PS: • Income (Frenette, 2008) • Financial literacy (Frenette and Robson, 2011) • Academic performance (Frenette, 2008) • Career information (Frenette, 2009)

  4. ReducingBarriers to Postsecondary: The Need for Hard Evidence • Although all four barriers are amenable to policy intervention, we have little to no hard evidence on best practices in Canada: • Students who qualify for grants or participate in academic / career counseling are likely a select group • Lack of natural experiments in Canada • Field experiments, using random assignment, are ideal, but rare…

  5. The Canadian Millennium Scholarship Foundation Recognized the Need for Hard Evidence (and they had hard cash!!!) • Commissioned the Social Research and Demonstration Corporation (SRDC) to design, implement, and evaluate three interventions to reduce specific barriers to PSE: • BC AVID • Academic intervention for middle-achieving students • Learning Accounts (Future to Discover) • Early promise of a large non-repayable grant for low-income students • Explore Your Horizons (Future to Discover) • Career / education information

  6. AVID • Advancement Via Individual Determination (AVID) • Serves 400,000 students in 4,500 schools worldwide • Goal: raise academic performance, and subsequently, university attendance • Targets students on the margin • Mostly Bs and Cs (middle-achieving) • No extreme behaviour issues • Desire to attend university

  7. AVID • What is AVID? • Academic intervention • Elective class (replacing a regular elective) offered throughout high school • Consists of: • Curriculum studies (40%) • Tutorials assisted by local college students (40%) • Motivational activities regarding PS (20%)

  8. AVID • Mechanisms through which AVID may help students (Dunn et al., 2008)? • Study skills (time management, note taking, etc.) • “Untracking” students (students choosing advanced courses) • Mentoring effects (continued contact with AVID teacher and tutors) • Peer effects (continued contact with students sharing similar characteristics)

  9. AVID • Can AVID make a difference? • Cognitive skills are not malleable after age 14 (Heckman, 1995), when AVID begins • But AVID helps students use their existing cognitive skills more efficiently by helping them become better learners • Furthermore, non-cognitive skills (motivation, self-discipline) are controlled by the prefrontal cortex, which is malleable until late adolescence (Heckman, 2000; Shonkoff and Phillips, 2000)

  10. AVID • Non-experimental evidence on AVID’s effectiveness: • Mehan et al. (1996): • Improvements in college participation • Watt et al. (2006) • AVID districts in Texas saw gains in graduation rates, advanced course enrollments, and international baccalaureate testing • Watt et al. (2007) • AVID participants had higher aspirations, knowledge about college, and academic preparation compared to peers • Nagaoka and LaForce (2010) • Propensity score matching study in Chicago • Small improvements in English and Math in grade 9; fewer absences • All suffer from selection bias • Those most likely to benefit from AVID will sign up

  11. BC AVID • First (and only) scientific evaluation of AVID • 1,241 students recruited at 14 BC high schools* • Two cohorts (2005 and 2006) followed throughout HS and until university age • Administrative and survey data used for follow-up • ‘AVID eligible’ students recruited through rigorous process: • Middle-achieving (mostly Bs and Cs) • No extreme behaviour issues • Desire to attend university *Schools and students had to apply to participate and waiting lists were created when classes reached the limit of 30 (just like the real AVID). Informed consent was also required from the parents to collect data.

  12. Evaluation approach • Random assignment within high schools • Impact: • Treatment group outcome – Control group outcome • Controlling for baseline characteristics

  13. Focus of this study • Impact of the offer of AVID on highest level of education aspired*: • Early demand indicator • Future work will report on university / college application / attendance * “No matter what you plan to be doing in a year from now (grade 12), what is the highest level of education you would like to get?”

  14. Large impact on university aspirations for boys and first-generation students Weaker AVID skills to begin with?

  15. No impact on non-university PS, as expected

  16. AVID may need more time to work… -

  17. Potential biases/evaluation challenges Those we can rule out • Program take-up • 96.7% accepted AVID offer • Spillover of AVID techniques • Very minor issue based on earlier comparison with similar non-AVID schools (Cornell notes) • Attrition bias (survey data analysis only) • 19.6% (treatment) vs 23.8% (control) between grade 9 and 12 • No important changes in baseline characteristics • Teacher grading bias • AVID students are well known • Argument does not apply to differences in sub-group impacts

  18. Potential biases/evaluation challenges Those we can not rule out • Program drop-out • 37.6% withdrew by grade 12 (mostly in that year: wanted to pick other courses to apply skills) • Estimated impacts = Intention-to-treat effects • Substitution bias (displacing non-experimental treatment) • Earlier report found reverse: course load was more challenging among treatment group • Timing • Lagged effect • Students coped with more difficult course load as they were only beginning to learn AVID skills • AVID now being run before HS in some jurisdictions

  19. Potential biases/evaluation challenges Those we can not rule out • Reactions to inequity caused by randomization • Positive or negative impact on control group • Currently following outcomes of similar students in matched non-AVID schools • Sample size • Affects statistical significance (especially in sub-group analysis)

  20. BC AVID Final Report • Final report (including cost-benefit study) in 2013

  21. Future to Discover Overview • Two interventions designed to increased PSE access: • Learning Accounts • Early promise of non-repayable grant (up to $8,000) for low-income youth • Offered in NB • Explore Your Horizons • Career and PSE planning intervention • Several components (workshops, magazines, parental outreach, etc.) • Offered in NB and MB

  22. How Can Learning Accounts Help? • Reduces cost of post-secondary, like other grants (clawed-back from loans) • Potentially addresses several design issues with existing grants: • Students are informed of the grant • Early promise of aid • No obligation to apply for loans • May encourage loan averse students to seek grant • Palameta and Voyer (2010) • Lab experiment: 5-20% students rejected grants when couple with a loan offer, but accepted same grant without loan offer (even though loan could be forfeited without compromising grant) • With no loans, Learning Accounts represents additional aid

  23. How Can Explore Your Horizons Help? • Informs students of costs and benefits of PSE • Financial literacy of students very poor regarding PSE (Frenette and Robson, 2011) • Helps students understand educational requirements for the careers they wish to pursue • Knowledge of educational requirements an issue for many students (Frenette, 2009)

  24. Methodology • Students are randomly assigned to receive the intervention (treatment group) or not (control group) • Three possible interventions: • LA • EYH • LA + EYH

  25. No PSE credential

  26. Why did Learning Accounts work in Francophone sector? • Possible explanation: supply constraints • Data from New Brunswick government: • Anglophone programs were oversubscribed • Francophone programs were undersubscribed

  27. Potential biases/evaluation challenges • Program take-up/drop-out • Everyone received offer of LA, but about 30% did not remember a couple of years later • Attendance in EYH workshops was poor, especially in Manitoba • Non-mandatory, after school • Spillover effects • Not an issue for LA • EYH workshops held after class • Substitution bias (displacing non-experimental treatment) • Minor impact in LA (small reduction in non-repayable aid for some groups) • EYH: looking into it • Attrition bias • Enrolment numbers use administrative data

  28. Potential biases/evaluation challenges • Reactions to inequity caused by randomization • Positive or negative impact on control group • No mechanism for testing this (did not follow non-experimental schools) • Sample size • Affects statistical significance (especially in sub-group analysis)

  29. Future to Discover Final Report • Final report (including cost-benefit study) in September 2012

  30. Appendix: Related academic interventions • QOP (Hahn et al., 1994; Rodriguez-Planas, 2010) • Includes financial incentives • IHAD (Kahne and Bailey, 1999) • Includes early promise of ‘last dollar scholarship’ • Career Academies (Kemple and Willner, 2008) • Focused on career themes • GEAR UP • Upward Bound (Seftor et al., 2009) • Upward Bound Math-Science (Seftor and Calcagno, 2010) Treatment broadly similar to AVID, but focus on low-income students

More Related