1 / 40

How much are we learning? Advances in school evaluation in New York and São Paulo

How much are we learning? Advances in school evaluation in New York and São Paulo. May 19, 2010. Jesse Margolis. Instituto Braudel. How much are we learning?. General observations about São Paulo schools Shared characteristics with New York Notable differences

mabyn
Download Presentation

How much are we learning? Advances in school evaluation in New York and São Paulo

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. How much are we learning? Advances in school evaluation in New York and São Paulo May 19, 2010 Jesse Margolis Instituto Braudel

  2. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 2

  3. Schools in São Paulo share challenges with New York • Both states enroll a large number of public school students… • New York State has 2.7 million students enrolled in 4.6K public schools • São Paulo State has 7.8 million students enrolled in 11.8K public schools (approx. 10 Federal, 5.7K State and 6.1K Municipal) • …and contain big cities with a substantial share of public school enrollment • New York City has 1.0 million students enrolled in 1.5K public schools • The city of São Paulo has 2.0 million students enrolled in 1.6K public schools (approx. 2 Federal, 1.1K State and 500 Municipal) • In both cities, a significant share of school-age children are enrolled in private schools… • In New York City, 19% of all students are enrolled in private schools, compared with 10% in the rest of the state and 10% in the entire United States • In the city of São Paulo, 18% of all students are enrolled in private schools, compared with 14% in the rest of the state and 12% in all of Brazil • …leaving the public school system primarily for low-income families • In New York City, 66% of public school students are eligible for free lunch due to low family income • In São Paulo, 68% of public school parents in the State system report an income of R$1,275 (about US$700) or less per month 3 Source: NYSED, Censo Escolar 2008, SARESP Cuestionário 2008; Note: excludes “eucação infantil” for São Paulo and PK for NYC.

  4. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 4

  5. Responsibilities for school governance are divided differently in São Paulo and New York São Paulo: State and municipal governments both regulate and run schools New York: State regulates school system and city runs schools 2.5MM 2.5MM 1.5MM • State: regulation and oversight • Establish standards • Certify teachers • Evaluate schools and districts • Define physical boundaries and governance of school districts • Redistribute funding between wealthier and poorer school districts • City: run schools • Build buildings • Hire principals, teachers and other staff & set salaries • Develop an admissions system for students • Decide on details of curriculum & professional development • Pay staff and other expenses MunicipalSchools Percentage of Students StateSchools 5 Source: Censo Escolar 2008

  6. Teachers in New York State earn above the median salary, but below most other professions requiring a college degree Average Salary ($US / year) US$ 65.899 / year Note: includes categories 25-2031, 25-2022 and 25-2021 for high, middle, and elementary school teachers. Does not include kindergarten, pre-school, or special education teachers Source: Occupational Employment Statistics (OES) Survey, May 2008. Bureau of Labor Statistics and Department of Labor 6

  7. In São Paulo State, the same is true, though salaries across the board are lower Average Salary ($R / month) 7 Source: PNAD 2008

  8. Among professions in São Paulo with a high share of university degrees, elementary and secondary teachers are the largest and lowest paid group Average Salary ($R / month) Note: includes only those professions where 80% or more of respondents said that their last level of education studied was “Superior – graduação” or “Mestrado ou doutorado”“Source: PNAD 2008 8

  9. Full-time teachers in São Paulo spend a larger share of their contractual time teaching students In or out of school (not teaching) In school (not teaching) In school (teaching) 41 periods (34hr 10min / week) 41 periods (34hr 10min / week) 40 periods (33hr 20min / week) Lunch/Transition (6 periods) Lunch/Transition (6 periods) Prep. (4 periods) HTPC* (3 periods) Professional Activity (5 per.) Professional Activity (5 per.) Preparation (5 periods) Preparation (8 periods) Teaching periods (50 minutes) Teaching (33 periods) Teaching (22 periods) Teaching (25 periods) * HTPC is shared planning and professional development timeSource: NYC DOE teachers contract (2007-2009); DECRETO Nº 55.078, DE 25 DE NOVEMBRO DE 2009 9

  10. 46% of public school teachers in the São Paulo State system report working in two or more schools Three or more Two Schools Percentage of Teachers One School Respondents: 7,102 15,874 17,542 11,763 10 Source: Qúestionario SARESP 2008

  11. Across all grade levels, average class sizes in São Paulo are larger than in New York São Paulo State (Avg. = 34.7)(Urban schools) New York City (Avg. = 26.1) Average Class Size Grade Level Note: Sao Paulo class size is for Regular Education in Urban, State Schools. New York City class size is for General Education and Gifted & Talented Source: 2008 Censo Escolar for São Paulo; 2009/10 Class Size Report for New York City 11

  12. Teacher absenteeism is two and a half times greater in São Paulo State than in New York City Recent Policies in São Paulo Have Focused on Teacher Attendance Average Teacher Absence Rate • Teacher Bonus: beginning in 2008, teacher performance bonuses of up to 22% of salary are proportional to their attendance (67% attendance minimum) • Teacher Raises: beginning in 2010, teachers must meet attendance minimums to be eligible for raises of 25% • “I don’t even think of going on strike, because if I am absent, I lose the one chance of earning a real raise” • São Paulo Public School Teacher, March 9, 2010. Percent of School Days Source: São Paulo figure is based on the 2008 school year from Veja interview with Paulo Renato Souza on 10/24/09. New York City figure is from Chicago Tribune article “Teachers miss days; poor kids miss out; Educators at some struggling schools take most time off, analysis shows, September 25, 2006.Teacher quote from “Paralisação dos professores em SP tem baixa adesão”, Estado de São Paulo, March 9, 2010 12

  13. More high school students in São Paulo go to school at night Percent of High School Students Enrollment: 981,767 132,593 331,437 13 Source: Censo Escolar 2009 (Sinopse Table 2.16); DOE ATS System as of 4/02/10

  14. Other observations about students • Students in São Paulo go to school for fewer hours • The school year is longer (200 days in São Paulo vs. 185 days in New York) • The school day for the student is shorter (4 hours to 5 hours and 20 minutes in São Paulo vs. 6 to 7 hours in New York) • Students in São Paulo take more courses at the same time • São Paulo: students are normally enrolled in 10 to 13 courses • New York: students are normally enrolled in 5 to 7 courses 14

  15. Middle and High School Teachers in São Paulo have a substantially higher student load than in New York • Students in São Paulo • go to school for fewer hours each week • and study more subjects at the same time • Teachers in São Paulo • teach more hours each week • have larger class sizes • and are more likely to teach in multiple schools •  Teachers have a much higher student load •  Teachers need to know the names and track the progress of many more students 15

  16. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 16

  17. New York: Accountability Framework 17

  18. Progress Report guiding principles The Progress Report was designed with several guiding principles in mind: • Outcomes, not inputs: Focus the conversation on student outcomes rather than inputs • Performance and Progress: Measure performance and progress of individual students • City and Peer Comparison: Compare schools to similar schools as well as a citywide standard • Simple, Valid, Verifiable: Ensure that schools can verify and re-create metrics, so schools understand how they are measured and how they can improve their performance 18

  19. Progress Report measures School Environment 15 points Additional Credit Up to 15 points Student Progress 60 points Student Performance 25 points Elementary, Middle, and K-8 Schools • Exemplary progress on test scores with high need students • Individual student progress on ELA and Math test scores • Student test scores in ELA and Math (median proficiency and % Level 3/4) • Learning Environment Survey results • Attendance High Schools • Graduation rates (4-year and 6-year) • More points for diplomas indicating college-readiness • Learning Environment Survey results • Attendance • Exemplary progress in credit gains with high need students • Credit accumulation • Regents completion and pass rates 19

  20. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 20

  21. SARESP is a good instrument to measure student learning • Performance Scale: Uses Item Response Theory (IRT) to assign each student a score for each subject, based on a scale from 0 to 500, with the majority of students between 125 and 400 • Proficiency Levels: In each subject, the scale is divided into four levels: • Advanced • Adequate • Basic • Below Basic 21

  22. IDESP is a simple way to combine two different objectives: improve student performance and improve promotion rate • IDESP is based on two indicators: • Performance Indicator: Based on the proficiency levels of students on standardized Portuguese Language and Math tests • Promotion Indicator: Grade promotion rate of the students Performance Indicator Promotion Indicator IDESP = X 22

  23. IDESP has a well thought out system of long and short-term targets • Long term targets (2030): The São Paulo State Education Department (SEE-SP) established long-term goals based on simulating IDESP results for developed countries in the OECD • School-specific targets: SEE-SP established intermediate targets for each school, based on the distance between its starting point and the long-term goal 23 Source: IDESP Technical Note, Secretaria Estuadual de Educação do Estado de São Paulo

  24. IDESP is relatively simple to calculate, though the State’s current explanation could be improved • The Performance Indicator (ID) isbased on the proficiency levels • Av = percentage of students at the advanced level • Ad = percentage of students at the adequate level • B = percentage of students at the basic level • AB = percentage of students below basic (abaixo do básico) A. Two-step formula described in the Nota Tecnica de IDESP Step 1: Calculate “Desfasagem” Step 2: Calculate Performance Indicator B. Equivalent one step formula IDjs = (10 x Av) + (6,66 x Ad) + (3,33 x B) + (0 x AB) 24 Note: an alternative formula for “B” would be IDjs = (10 x Av) + (20/3 x Ad) + (10/3 x B) + (0 x AB)

  25. Positive: IDESP is given strong teeth by the bonus program Maximum Bonus as a Percent of Average Annual Teacher Salary Percent of Annual Salary Fonte: New York City bonus is $3,000 per teacher provided to schools in a 200-school pilot project that meet Progress Report targets. São Paulo bonusis up to 2.9 additional monthly salaries for teachers whose schools improve their IDESP scoreSource: teacher salaries based on PNAD 2008 for São Paulo and Bureau of Labor Statistics May 2008 Occupational Employment Statistics survey 25

  26. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 26

  27. Change in IDESP from 2007 to 2008 Note: each school is represented by a separate point. Source: Boletim IDESP 27

  28. Change in IDESP from 2008 to 2009 Recommendation: Interview school personnel to identify best practices and review testing micro-data to identify potential weaknesses in the reliability of test score and passing rate indicators. Note: each school is represented by a separate point. Source: Boletim IDESP 28

  29. There is no incentive for weaker students to take the test • “There were many schools that only sent the good students, and if you only send the good students, your IDESP is better. The great failure of IDESP is that it does not consider the number of students who take the test” -- High School Principal, São Paulo State Average Test Taking Percentage (SARESP 2008) Test Taking Percentage by School (SARESP 2008, Grade 9) Recommendation: incorporate percentage of students taking the test into IDESP, potentially by multiplying the test taking percentage by the IDESP score 29 Source: Boletim SARESP 2008; Censo Escolar 2008; Author interviews.

  30. Individual student or item results are not provided to the schools, limiting SARESP’s use in improving education “SARESP is not a tool for improving education...we don’t have access to the student test results” - High School Teacher, São Paulo State “Who are the students who are below basic; we don’t know What did they get wrong; we don’t know...How am I going to work with the students this year if I don’t know what they got wrong” - High School Curriculum Director, São Paulo State Recommendation: Develop a plan to provide individual test results to schools and parents. 30 Source: author interviews

  31. The parent survey asks many questions that can paint a picture of a school’s learning environment • Parents: To what degree to you agree or disagree with the following statements: • Communication with parents • 2.1. I receive information from the school about my child’s progress. • 2.11. The school always holds parent meetings to inform me about my child. • Parent Involvement • 2.17. When there is a problem I am quickly called to the school. • 2.19. The school recognizes that parents’ opinions are important. • Student Safety • 2.3. My child is safe in school. • 2.16. My child feels safe in school. • 2.23. This school has many students with behavioral problems. • Opinion about teachers • 2.4. The school’s teachers respect the students. • 2.12. I believe the teachers are very capable. • Opinion about the school • 2.15. I would like my child to study in another school. • 2.24. If I could pay, my child would study in private school. Note: parent survey is given once per year to parents of all students who take the SARESP (5th, 7th, 9th, 12th grades)Source: SARESP 2008 Questionário dos pais 31

  32. The survey could give valuable system-wide information for the State of São Paulo 1,174,660 1,119,737 1,130,132 1,195,661 1,136,501 Disagree Partly Agree Percentage of Respondents Completely Agree 32 Source: SARESP 2008 Questionário dos pais

  33. The survey could provide parents, teachers, and students with an insider’s perspective into school quality Percentage that Completely Agree São Paulo State School A School B Recommendation: publish results of parent, teacher, and student surveys, aggregating results to maintain confidentiality 33 Source: SARESP 2008 Questionario dos pais

  34. NYC Simulation: On average, schools with better Progress Report letter grades had higher simulated IDESP scores in 2009 Simulated IDESP (2009) N* = 2 11 138 886 Note: Includes Ensino Fundamental I (4th grade) and Ensino Fundamental II (8th grade) separately, so schools with both grades may appear twice in the chart. 34

  35. IDESP Simulation: However, there was substantial variation around these trends Progress Report Overall Score (2009) R = 0.20 Simulated IDESP Score (2009) Note: Includes Ensino Fundamental I (4th grade) and Ensino Fundamental II (8th grade) separately, so schools with both grades may appear twice in the chart.

  36. Hypotheses to explain difference between IDESP and NYC Progress Reports • The two instruments measure different things • NYC Progress Reports include parent survey as 10% of the grade • IDESP includes student grade promotion rates • NYC Progress Reports include more students • IDESP only includes the last year of elementary school and last year of middle school • Progress Report includes all students that take the test: 3rd – 5th in elementary school and 6th – 8th in middle school • 75% of the NYC Progress Report is based on a comparison between similar schools; IDESP is calculated equally for all schools in the state. • The NYC Progress Report measures students’ individual progress; the change in IDESP compares the results of one group of students in a particular grade one year with a different group of students in the same grade the following year • Recommendations: Consider “Additional Credit” for: • Individual student progress • Performance relative to similar schools 36

  37. Recommendation: Continue developing tools to help principals, teachers, and other school professionals do their job better Survey with Teacher Coordinators (April, 2010) IDESP Modeler n = 37 • Excel tool that allows the school to simulate student test performance and grade promotion results and see the impact on target achievement and teacher bonus Completely disagree Disagree Percent of Respondents Agree Completely agree 37

  38. How much are we learning? • General observations about São Paulo schools • Shared characteristics with New York • Notable differences • Brief presentation of New York City’s accountability system • School evaluation in São Paulo • Best practices • Opportunities for improvement • Recommendations 38

  39. Recommendations • Short-term • Simplify the explanation of IDESP and implement active training effort so that school staff understand the calculation • Further study why certain schools had large jumps in IDESP to identify best practices or test-security weaknesses • Incorporate the percentage of students taking the test into the calculation of IDESP • Provide school-wide parent, teacher and student survey results to the public, aggregating results at the school level to maintain confidentiality • Long-term • Develop a plan to provide individual student test results to schools and parents • Consider incorporating individual student progress and performance relative to similar schools into the measure • Continue developing tools to help school principals, teachers and other staff do their job better 39

  40. Thank you! Jesse Margolis jesse@post.harvard.edu 40

More Related