1 / 36

California's API: Recent Developments and Future Prospects

California's API: Recent Developments and Future Prospects. Edward H. Haertel. Stanford University School of Education CRESST Conference University of California at Los Angeles September 10, 2004. The "API" is California's Academic Performance Index.

tracey
Download Presentation

California's API: Recent Developments and Future Prospects

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. California's API:Recent Developmentsand Future Prospects Edward H. Haertel Stanford UniversitySchool of Education CRESST Conference University of California at Los AngelesSeptember 10, 2004

  2. The "API" is California'sAcademic Performance Index • School-level summary of student test performance • Mandated by Public Schools Accountability Act of 1999 (PSAA) • Foundation of California's (pre-NCLB) public-school accountability system

  3. Plan for This Morning • Calculating the API • Coping with change • Lessons learned

  4. Calculating the API

  5. Calculating the API in 1999It used to be so simple... • Index based just on SAT-9 scores • Elem/Middle: Reading, Language Arts, Spelling, Math • High School: Reading, Language Arts, Math, Science, Social Studies

  6. Reading 30% Lang Arts 15% Spelling 15% Math 40% Reading 20% Lang Arts 20% Math 20% Science 20% Soc St 20% Subject Area Weights Elem/Middle High School

  7. 1-19 20-39 40-59 60-79 80-99 200 500 700 875 1000 Precursors to Standards-Based Performance Levels SAT-9 National Percentile Rank "Progressive Weighting Factor"

  8. Far Below Basic Below Basic Basic Proficient Advanced 200 500 700 875 1000 Performance Levels on California Standards Tests (CSTs) CST Performance Level "Weighting Factor"

  9. Growth Targets • Statewide Perf. Target = 800 • Annual Growth Target =5% of distance from Base-Year API to 800 • Initially, schools at/above 800 need only remain at that level

  10. Numerically Significant Subgroups • "Comparable Improvement" require-ment mandated by PSAA of 1999 • Groups defined by Race/Ethnicity and by socioeconomic disadvantage • Student in soc. dis. group if either(1) eligible for free/reduced price meal program or (2) highest parent educational level is "not a high school graduate."

  11. Defining "Numerically Significant" • How large must a subgroup be to count as numerically significant? • Fewer than 30 students not num. sig. • 30-99 students num. sig. only if subgroup constitutes at least 15% of total enrollment • 100 or more students are num. sig. • For purposes of API growth calculation, subgroup must be num. sig. in both base year and growth year

  12. Defining "Comparable Improvement" • Growth target is defined in terms of difference between subgroup's score in base year and in growth year • Subgroup growth target is 80% of schoolwide target

  13. Coping with Change

  14. "IF YOU WANT TO MEASURE CHANGE, DON'T CHANGE THE MEASURE." --Al Beaton

  15. Base API and Growth API Biannual Accountability Cycle May ... Oct ... Feb ... May ... Oct ... Feb ... testing Growth Base testing Growth Base API API API API Release Release Release Release testing cycle testing cycle School Year School Year

  16. The "Scale Calibration Factor" • The SCF is a constant added to both the Base API and the Growth API within an accountability cycle. • It is chosen to make the mean Base API for that cycle equal to the mean Growth API for the previous cycle. • There are new SCFs each year for Elementary schools, Middle schools, and High schools

  17. The "Scale Calibration Factor" Adjusted to equal state-level means via SCF Adjusted to equal state-level means via SCF May ... Oct ... Feb ... May ... Oct ... Feb ... testing Growth Base testing Growth Base API API API API Release Release Release Release testing cycle testing cycle School Year School Year

  18. Changes in the Measure • Changes in the tests included • Changes to accommodate heterogeneity of California schools • Evolving decision rules for special situations • Changes due to NCLB Act of 2001 • Refinements in data acquisition/ quality control

  19. Changes in the Tests Included • 1999-00 SAT-9 only • 2000-01 SAT-9 only • 2001-02 add California Standards Test (CST) in ELA • 2002-03 add CST-Math (G2-11); add CST-Hist/SocSt (G10- 11); drop SAT-9 Soc St; add CAHSEE G9-10

  20. Changes in the Tests Included • 2003-04 add CST-Science (G9-11); incorporate CAPA; CAHSEE now G10-11; add CAT-6; drop SAT-9 • 2004-05 ??? (2004-05 Base API to be posted March 2005)

  21. Changes in the Tests Included Future changes will include: • reducing CAT-6 to brief survey at just two or three grade levels • adding CST-Science (G5) • adding cumulative Hist/Soc Sci (G8) • expansion of CAHSEE to include G11, G12 retests • ... ?

  22. Heterogeneity of California Schools • Schools that cut across usual Elem/Middle/High boundaries • Small and very small schools • Alternative School Accountability Model (ASAM) schools

  23. Most Common Grade Spans for California Schools Data from "Accountability Update – Spring 2004" at http://www.cde.ca.gov/ta/ac/ay/

  24. Small (and Very Small)and ASAM Schools • First, schools with N < 100 did not receive API • Then, schools with N between 11 and 99 received API with an asterisk • Now, due to inclusion of API as alternative indicator in California's NCLB accountability, all schools must receive API, including ASAM schools and those with N < 11

  25. Evolving Decision Rules • denominator (total enrollment) for testing participation rate calculation • Parent opt-outs • students tested with • Testing accommodations • Testing modifications • California Alternate Performance Assessment (CAPA)

  26. Evolving Decision Rules • Testing irregularities • Student testing irregularities • Adult testing irregularities • Decision rules for deciding which has occurred • Appeals processes • Growth calculations when API is invalidated for one year

  27. Evolving Decision Rules • Mobility Exclusions • Students new to district • Students new to school • Special rules for "feeder school" patterns • Non-universal indicators • High School Students not enrolled in a math or science course • Changing rules for out-of-level testing

  28. Evolving Decision Rules • Partial records • Counting students taking some but not all subtests • G4 or G7 writing tests with no CST-ELA • blank test forms • Students taking high school math or science courses earlier/later than standard grade sequence

  29. Impact of NCLB... ... has been massive, requiring changes in what subgroups are included, how participation rate is calculated (exclusions, averaging), what schools get APIs, what tests are given at each grade, rules for out-of-level testing, data release timelines, design of score reports, treatment of students designated proficient on CAPA, ...

  30. Data Acquisition/Quality Control • School-level analog to standardized testing conditions at student level • Requires looking at API as product of complex system, dependent on • Government officials, State Board of Ed • Actors in school and district offices • Testing contractors • Teachers

  31. Data Acquisition/Quality Control • Data sources have changed over time for • Total school enrollment numbers • Free/Reduced Price Meal Participation numbers • New "2004 STAR Pre-Identification Data Review"

  32. Lessons Learned

  33. Lessons Learned • Long-term stability is probably unattainable • Complexity increases, and increases, and increases • Technical and policy considerations often point to divergent courses of action

  34. Lessons Learned • Seeing Like a State* • Fairness is a matter of degree • Comparison of API (successive cohort) vs. matched individual growth indices • Inequities in accounting for educational challenge • Paradoxical effects *Title of 1998 book by James C. Scott

  35. Lessons Learned • With hard work, a system can be created that is trusted and respected, despite its imperfections • By and large, scores improved steadily over the entire time California used the SAT-9* • An enormous "behind the scenes" effort is required to maintain a reliable and valid accountability system *See David Rogosa's paper, "Four-peat: Data Analysis Results from Uncharacteristic Continuity in California Student Testing Programs" at http://www.cde.ca.gov/ta/ac/ap/researchreports.asp

  36. What Can Policy Makers Do? • Create and use a stable technical advisory committee • Plan ahead, but stay flexible • Resist pressures to use tests for new and different purposes • Remember that the school-level index is only part of an accountability system and only part of an education indicator system

More Related