1 / 20

New York City Department of Education

New York City Department of Education. Beat-the-Odds HS Update. DRAFT WORK PRODUCT For Discussion Only. March 6, 2008. Beat-the-Odds HS Analysis Update Objectives for Today. Review structure and initial findings from updated regression model

arkadiy-popov
Download Presentation

New York City Department of Education

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. New York CityDepartment of Education Beat-the-Odds HS Update DRAFT WORK PRODUCTFor Discussion Only March 6, 2008

  2. Beat-the-Odds HS Analysis UpdateObjectives for Today • Review structure and initial findings from updated regression model • What does this model allow us to do, and what questions do we want to ask? • Discuss data on specific follow-up questions from last data review • Gather group input on key areas of potential implications: which findings should be highlighted, and which applications of the data are most powerful? • Discuss key areas of analytical next steps: • Variables to test which might be common across Beat-the-Odds HS’s • Secondary portfolio strategies to model and assess projected impact

  3. How Does the New Regression Model Work? Student-Level Factors School-Level Factors True Over- and Under-Performance by Schools • For each individual student, what are the key demographic and performance variables that shape the odds of graduation? • Given each student’s particular profile, what is each individual’s expected graduation rate? • What are the aspects of school structure that have an additional “peer effect” on the chances of graduation? • By how much is each individual student’s odds of graduation changed by attending schools with different structures and populations? • Once we have leveled the playing field in terms of both student-level and school-level factors, how much variation remains in the graduation rate across schools? • Which schools are the greatest over- and under-performers, and are there factors that appear to enable such performance?

  4. What Are the Variables that Drive the New Model? Student-Level Factors School-Level Factors True Over- and Under-Performance by Schools • Gender • Ethnicity (White/Asian vs. African-American/Hispanic) • Average 8th ELA + Math Proficiency Level • Age at Entry to High School • 8th grade attendance rate • ELL Status (Y/N) • SPED Status (LRE vs. Self-Contained vs. Non-SPED) • Lack of 8th grade attendance or 8th grade test scores • School enrollment • A series of concentration variables: • Percent of students with a Low-Level 2 or below on either ELA or Math • Percent of students entering 9th grade overage • Percent of students with 8th grade attendance below 90% • Tested but not significant: Percent ELL, Percent SPED (total and Self-Contained only), Title I eligibility, Avg. teacher tenure, Principal tenure, General Ed class size, Special Ed class size, facilities capacity utilization • The model explains 71% of the variation in high school graduation rates • However, 22% of schools differ from their predicted graduation rate by +/- 10 or more percentage points

  5. Individual Student Odds of Graduation Are Predicted Based on a Combination of Factors • Note: Predicted graduation rates below are calculated independent of the schools that students attend Predicted Graduation Rate byStudent-Level Factors, 2007 Cohort Number of Factors Related to High Graduation Rate Many Few Ethnicity: Wh./As. Af-Am/His. Af-Am/His. Af-Am/His. Af-Am/His. Af-Am/His. Af-Am/His. Af-Am/His. Gender: Female Female Male Male Male Male Male Male Test Level: 3.0 3.0 3.0 2.0 2.0 2.0 2.0 2.0 Age at Entry: On On On On Over Over Over Over 8th Gr. Att.: 95% 95% 95% 95% 95% 85% 85% 85% SPED Status: GE GE GE GE GE GE LRE SC Note: Controlling for all other factors, students who are ELL during high school are 9% more likely to graduate than non-ELL students Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  6. White/Asian Females White/Asian Males African-American/Hispanic Females African-American/Hispanic Males The Model Can Isolate the Effect of Demographic Variables on Student Performance • At the widest point (low-mid Level 2), white and Asian females outperform African-American and Latino males by more than 20% points – controlling for all other factors • Note: Predicted graduation rates below are calculated independent of the schools that students attend Predicted Graduation Rate by Ethnicity and Gender, 2007 Cohort Average of 8th Grade Math & ELA Proficiency Rating Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  7. SPED Students in CTT Environments Appear to Outperform Peers in Both LRE and Self-Contained • However, at higher levels of incoming proficiency, CTT students still under-perform general ed students Credit Performance of Special Education Populations byIncoming 8th Grade ELA Proficiency Level, First-Time Freshmen, 2006-07 Non-Special Ed. 76% 57% 44% 31% Number of Students Note: Districts 75 and 79 excluded; Excludes students with missing 8th Grade ELA proficiency data Source: DOE Internal Data

  8. School Structure Can Also Drive Wide Changes in Expected Performance • School 1(e.g. Science, Tech and Research Early College HS) • 400 students • Concentrations of challenged students 15% points below system average • School 2(e.g. HS for Leadership and Public Service) • 800 students • Concentrations of challenged students 5% points below system average • “The Median Student” • Female • African-American/Hispanic • Median 8th Grade Test Scores (2.7) • Med. 8th Grade Attendance (92%) • On-Age at Entry to HS (14 yo) • Neither ELL nor SPED 83% Graduation Rate 78% Graduation Rate (Effect of 400-StudentEnrollment Alone  78%) • School 3(e.g. William Grady CTE HS) • 1,500 students • Concentrations of challenged students 5% points above system average • School 4(e.g. John F Kennedy HS) • 3,000 students • Concentrations of challenged students 15% points above system average 70% Predicted Graduation Rate 70% Graduation Rate 55% Graduation Rate Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  9. African-American/Hispanic Male • 8th Grade Test Score of 2.0 31% Predicted Graduation Rate(independent of school attended) • Improve 8th Grade Test Score to 2.5 • Attend a 3,000 Student School with Low Achievement, Overage & Low-Attending Student Concentrations 10 p.p. Above System Average • 8th Grade Test Score Remains at 2.0 • Attend a 400 Student School With Average Concentration of Low Achievement, Overage & Low-Attending Students 40% Graduation Rate 42% Graduation Rate The Model Allows Us to Compare the Impact of Different Strategies on Expected Student Performance Improve Preparation in MS HS Portfolio Development Source: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  10. The New Model Highlights a Greater Number of Schools with Low Predicted Grad Rates Relationship Between Predicted Graduation Rate and Actual Graduation Rate, 2007 Cohort Schools with theToughest Odds • Less than 40% predicted grad rate: 5 schools (vs. 0 in original model) • 40-45% predicted grad rate: 12 schools (vs. 3 in original model) • 45-50% predicted grad rate: 24 schools (vs. 14 in original model) R² = 0.71 # of Obs. = 230 New Small High Schools Traditional High Schools & Career/Technical Source: ATS Data; Progress Reports Data; NYC DOE Website

  11. Over-Performance(15% of schools) R² = 0.71 # of Obs. = 230 New Small High Schools Traditional High Schools & Career/Technical Underperformance(7% of schools) 22% of Schools Diverge from Their Predicted GraduationRate by +/- 10 or More Percentage Points • 78% of New Small High Schools have an actual graduation rate that exceeds their predicted graduation rate; 37% of New Small High Schools exceed this prediction by 10 percentage points or greater Relationship Between Predicted Graduation Rate and Actual Graduation Rate, 2007 Cohort Note: 10% points represents approximately one standard error in the final regression modelSource: ATS Data; Progress Reports Data; NYC DOE Website

  12. Performance of New Small Schools Differs from Small Schools Opened During the 1990’s Distribution of 2007 Over/Under-performance for New Small Schools (n=46) Average = 8.1% Residual Distribution of 2007 Over/Under-performance for Old Small Schools (n=62) Residual Average = -0.3% Source: DOE Internal Data; Parthenon Regression Model

  13. New Schools Outperform Most with Challenged Students, but Much of the Gap Is in Local Diplomas Class of 2007 Four Year Graduation Rate by 8th Grade ELA Level:New Small Schools vs. All Other Schools, Local vs. Regents Diplomas Level 1 Low Level 2 High Level 2 Level 3 Level 4 Overall Gap 18% 26% 17% 9% N/A Regents+ Gap 4% 7% 6% 3% N/A * Limited sample (n=49) for Level 4 students in New Small SchoolsNote: Only includes schools that meet requirements for regression analysis (i.e., 20-student minimum cohort size, progress report grade). Excludes “Other” diploma types from non-New Small Schools, including “Proof Of Receipt Of HS Diploma,” “Already Had Diploma, Tried For Regents But Left,” “Career Ed Regents Diploma,” “Confirmed Completion Of HS Requirements,” “Advanced Diploma CTE,” and “Local Diploma CTE” Source: DOE Internal Data

  14. Regression Is Run Over Time to Assess Performance Trajectory School Over- vs. Underperformance in New Regression Model, Class of 2005 vs. 2007 Under-Performance Over-Performance 2005 Cohort Regression Over-Performance • Of the schools who were underperforming in the Class of 2005, 82% were still underperforming in the Class of 2007 • Of the schools who were over-performing in the Class of 2005, 63% were still over-performing in the Class of 2007 • Of schools with graduating classes in both 2005 and 2007, 57% are under-performing in 2007 – driven by the disproportionate rate of over-performance among new small schools 2007 Cohort Regression Under-Performance Note: Percentages are taken of all schools that met requirements for regression analysis for both 2005 and 2007 cohorts; percentages may not total to 100% due to roundingSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  15. New Small Schools Appear Mixed in Sustaining Their Performance Going Forward • Overall, the percent of students earning 20+ credits by the end of the 2nd year went from 64% to 62% in the new small schools (vs. 49% to 54% in the rest of the portfolio) Change in Percent of Students Earning 20+ Credits After Two Years, 2007 vs. 2009 Cohort Biggest Improvement Belmont Preparatory HS (+13%) Fordham HS For The Arts (+11%) Jonathan Levin HS For Media And Communications (+10%) Fordham Leadership Academy For Business And Technology (+4%) Bronx HS For Medical Science (+3%) Biggest Decline School For Excellence (-26%) Frederick Douglass Academy II Secondary School (-22%) Community School For Social Justice (-19%) Bronx Leadership Academy II HS (-16%) Biggest Improvement HS For Violin And Dance (+20%) Brooklyn Academy Of Science And The Environment (+12%) Manhattan Bridges HS (+11%) Brooklyn HS For Music And Theater (+8%) New York Harbor School (+5%) Biggest Decline Pelham Preparatory Academy(-28%) International Arts Business HS (-23%) Discovery HS (-17%) Bushwick HS For Social Justice (-15%) HS For Contemporary Arts(-12%) Percent of New Small Schools Percent of New Small Schools More analysis would be required to adjust these figures for changes in school population Source: DOE Internal Data

  16. Three High-Performing Categories of Schools Present KM Opportunities Proposed Key Categories of High Schools Beat-the-Odds Schools Schools with“Pockets of Success” “Turnaround” Schools Proven Beat-the-Odds Emerging Beat-the-Odds • “Emerging Beat the Odds” Schools outperform in the most recent cohort (2007) by the same margin as Proven Beat the Odds schools (at least 5% points) • Also exceed citywide average graduation rate • These schools also show signs of sustaining their performance going forward: • Percent of 2nd year students with 20+ credits has improved or remained steady within 5 percentage points • “Proven Beat the Odds” Schools demonstrate a sustained ability to outperform their predicted graduation rate: • Exceed predicted graduation rate by at least 5 percentage points in both Class of 2005 and 2007 • Also exceed citywide average graduation rate • Schools that do not meet the criteria of “Beat the Odds” schools overall, but which perform as well as the best “Proven Beat the Odds” schools with key sub-groups • Groups tested thus far include: • Low achieving 8th graders • 9th graders who are overage at entry to HS • ELL students • Self-Contained SPED students • African-American and Latino males • Turnaround schools should demonstrate dramatic improvement over the period from 2005-2007: • Underperforming in 2005 and over-performing in 2007 • Improved by 10%+ points in both relative and absolute grad rates • Now have a graduation rate at or above the citywide average 17 schools(across 5 categories) 25 schools 21 schools 6 schools Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model

  17. Matching Schools with a Counterpoint Provides a Unique Opportunity for Improvement Proposed Key Categories of High Schools Beat-the-Odds Schools Schools withPockets of Success “Turnaround” Schools Proven Beat-the-Odds Emerging Beat-the-Odds Schools with ChronicUnderperformance Struggling New Schools Schools with SpecificAreas of Weakness Schools in Decline • New small schools that: • Underperform their predicted graduation rate by 5 or more percentage points or • Show substantial declines in credit accumulation between the Class of 2007 and Class of 2009 • Schools that underperform their predicted graduation rate by 5 or more percentage points in both the Class of 2007 and the Class of 2005 • Schools that are over-performing their predicted graduation rate by 5 or more percentage points, but lag the citywide average in a key student sub-group • Schools in decline demonstrate a dramatic deterioration in performance: • Went from over- to underperforming their predicted rate from 2005 to 2007 • Decline in performance relative to expectations of 10%+ percentage points TBD (In Process) 12 schools(excluding Phaseouts) 15 Schools 12 schools Note: Counts of schools in each category remain preliminary based on ongoing changes to regression model

  18. Beat-the-Odds High Schools Mostly Score Highly on Progress Reports and Quality Reviews Proven Beat-the-Odds Schools Emerging Beat-the-Odds Schools Over-Performed by 10% or Greater On Both Over-Performed by 10% for Class of 2007 School Name Grad Rate PR QR School Name Grad Rate PR QR Over-Performed by 5% or Greater On Both School Name Grad Rate PR QR Over-Performed by 5% for Class of 2007 School Name Grad Rate PR QR 12 out of 21 schools with either an A or W 19 out of 25 schools with either an A or W Note: In order to be considered for “beating the odds,” a school must exceed the citywide average graduation rateSource: ATS Data; Progress Reports Data; DOE Internal Data; NYC DOE Website

  19. Proven BTO Other Schools In Addition to Predicting Average Performance, Size and Concentration Appear to Enable Beat-the-Odds Performance • The chart below looks at school size in relation to percent of students who are overage at entry to HS • The same finding applies for large schools and above-average concentration of low-proficiency students (only Harry Truman HS is the only such school beating the odds) Relationship Between Total Enrollment and Proportion of Overage Students, 2007 Cohort 1 2 Legend: Emerging BTO Harry S Truman HS Percentage of Schools In Each Quadrant That Beat-the-Odds 3 4 34% 3% 1 2 3 18% 4 14% Note: Only includes schools that meet requirements for regression analysis (e.g., 20-student minimum cohort size, progress report grade) Source: DOE Internal Data

  20. Potential Areas of Implication PortfolioDevelopment HS Admissions KnowledgeManagement • What targets should be set for the overall evolution of the portfolio with respect to key structural variables? • Can we isolate the factors that have made the new school creation process more effective than the 1990’s school openings? • Should we consider constraints on the HS admissions process that take into consideration the predicted graduation rate of the school? (e.g. “don’t allow any school to have a predicted rate less than 45%”) • What processes and structures should be created that: • Allows on-site investigation to determine what drives beat-the-odds performance? • Enables beat-the-odds schools to codify their own practices? • Encourages collaboration among like schools with differential performance? • What is the nature of the questions we should be asking to define those inquiries? • What other implications emerge from today’s findings? • What additional questions should we be pursuing to inform the broader Secondary (or Middle School) strategy?

More Related