Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade Students' Learning and Stud

1. Evaluating the Impact of the Advancement Via Individual Determination Program on Ninth-Grade Students' Learning and Study Skills Jenny Nagaoka, Melissa Roderick, and Melanie LaForce June 29,2010 2010 IES Conference

2. What is AVID? • History: • Began in 1980 in San Diego, CA • National AVID office was created to provide standards and oversight in implementation and expansion • 2009-2010 school year: nearly 4,500 schools in 45 states • AVID in the Chicago Public Schools (CPS): • Part of a larger initiative to improve student preparation for and access to postsecondary education • First implemented in CPS in 2003-2004 in 8 high schools • Expanded to 42 high schools by 2007-2008 and 59 high schools currently, including 2 all-AVID high schools.

3. The AVID Program Program & Goals: • School-based program, provides academic and social support • Designed to improve the preparation and college access of middle-achieving students The AVID Elective Class- “How to be a student” Curriculum around Supports & Skill Building: • Academic – organizational, note-taking, writing, critical thinking skills, and questioning skills • Social – peer support, teacher guidance, college advising The AVID Site Team • Site Coordinator/Teacher; AVID Administrator; English, math, science, social studies teachers

4. Research on AVID • Some positive evidence • More likely to be enrolled in college (Mehan, et al., 1996 ) • Schools with AVID have higher graduation rates, advanced course-taking (Watt, et al., 2006) • But do not address selection effects • SRDC British Columbia Study • Randomized Experiment • Non-urban, non-minority district

5. Data and Sample • Data • Consortium on Chicago School Research data archive • CPS school records, including high school transcripts • Biennial survey of CPS students • Sample • First-time 9th graders in 14 CPS AVID high schools, not in special education, enrolled in both fall and spring semesters • 7,357 students in pre-AVID cohorts (2002 and 2003) • 14,031 students in post-AVID cohorts (2005 to 2008), 2,521 AVID students and 11,510 non-AVID students • 51% female and 49% male, 68 % African-American, 23% Latino, 10% White

6. Research Questions • Are AVID students doing better than other students on critical 9th grade outcomes that predict high school and college graduation? 2. Are these differences due to selection effects?

7. Does AVID have an impact on 9th grade outcomes? • Supports & Skill Building • Academic: study habits • Social supports: peer support, sense of belonging, teacher-student trust, teacher support, engagement • Academic outcomes • Cumulative weighted GPA and probability of 3.0 or higher GPA • On-track • Absences • English GPA and probability of B or better (English I) • Math GPA and probability of B or better (Algebra or geometry)

8. Dealing with Selection Bias 1. Propensity score matching • Estimate probability for treatment based on measured variables • Drawbacks: Only matching on measured characteristics (test scores, demographic characteristics, fixed effects for elementary school, 7th and 8th survey responses) • Sample of students in 14 AVID schools is divided into deciles based on propensity scores

9. Dealing with Selection Bias 2. Simulated Propensity Scores • Pre-post comparison of treatment constructed by simulating a comparison group of students who have a high propensity for AVID in the pre-AVID period • Apply propensity model coefficient weights and simulate probability for AVID in students before AVID was implemented • Three groups: pre, post-period AVID, post-period not in AVID • Pre- and post non-AVID should be similar once we control for background characteristics and cohort effects. • Sample is divided into deciles based on propensity scores with same cut points for pre and post cohorts.

10. Average GPA for 9th grade AVID and Non-AVID studentsPost-AVID cohorts 2003-2008

11. Probability of having a B or better GPAfor 9th grade AVID and Non-AVID studentsPost-AVID cohorts 2003-2008

12. On-track and absences for 9th grade AVID and Non-AVID studentsPost-AVID cohorts 2003-2008 9th grade on-track rates Days absent per semester in 9th grade

13. Allensworth & Easton (2007) What matters for staying on-track and graduation from high school in Chicago Public Schools

14. Propensity Score Matching9th grade academic outcomes for Post-AVID cohorts 2003-2008

15. Simulated Propensity Scores9th grade academic outcomes for Pre-AVID cohorts (2001 and 2002) and Post-AVID cohorts (2003-2008)

16. 9th grade study habits for Post-AVID cohorts 2003-2008 How much do you agree with the following: • I set time aside to do my homework and study. • I try to do well on my schoolwork even when it isn’t interesting to me. • If I need to study I don’t go out with my friends. • I always study for tests.

17. Propensity Score Matching 9th grade academic skill and social outcomes for Post-AVID cohorts 2003-2008

18. Conclusions • AVID is clearly doing a good job of selecting students • AVID students are performing poorly in school • Average GPA is <2.0 • Lack of Strong Effects may be linked to differences between CPS and other AVID districts • Implementation issues • “Middle Achieving” may mean different things • Teaching study skills may not translate to large urban districts • AVID focuses on improving student skills without changing what is happening in the classroom.