1 / 36

Powerful Data Tools: Moving Beyond Accountability

Powerful Data Tools: Moving Beyond Accountability. Venessa A. Keesler, Ph.D. Bureau of Assessment and Accountability Michigan Department of Education Presentation to MASFPS Fall Directors’ Institute October 4, 2012. Things I’ve learned about accountability since taking this job.

zudora
Download Presentation

Powerful Data Tools: Moving Beyond Accountability

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Powerful Data Tools: Moving Beyond Accountability Venessa A. Keesler, Ph.D. Bureau of Assessment and Accountability Michigan Department of Education Presentation to MASFPS Fall Directors’ Institute October 4, 2012

  2. Things I’ve learned about accountability since taking this job • #1: NO ONE (and I do mean no one) likes accountability. • #2: If you are the one who is holding people accountable, you are not going to be the most popular person in the room • #3: See point number 1

  3. So why do we do accountability? • Three myths; one reality • Myth #1: To drive reform • Myth #2: To create education policy • Myth #3: Because we are gluttons for punishment • Reality: • Accountability metrics/systems are quantitative articulations of the core policy beliefs of the education system • They help us measure our progress in meeting those core policy goals • They are the measure, not the purpose or the goal

  4. Accountability Landscape: 2012 • A new era of accountability • Switching from a purely criterion-based system to a normative system • Criterion-based systems: Set average proficiency targets for schools. • Normative system: identifies the “worst” or “best” or “lowest” or “highest”

  5. Why the change? • Policy imperative for NCLB: all students CAN and SHOULD demonstrate proficiency  criterion-system with proficiency targets for all schools and subgroups • 10 years later: our average achievement is increasing, but we still have students and schools lagging behind • New policy imperative (ESEA Flex): we must target our lowest performing schools AND our lowest performing students more specifically and strategically

  6. The problem with average proficiency rates at a school or district level • “Never trust an average” • Averages mask low performance • Example: • Proficiency target is 50% • Sunshine School has 20 students, 10 of whom are proficient and 10 of whom are not. • Sunshine School meets it’s target; hooray! • BUT  those 10 not proficient students are possibly left behind

  7. Michigan’s Accountability System • Top to Bottom Ranking • Priority Schools (bottom 5%) • Focus Schools (largest achievement gaps) • Reward Schools (high performing, high progress, beating the odds) • Accountability Scorecard • Proficiency targets for all schools

  8. Moving Beyond the Label • We believe: • Accountability only measures the core policy beliefs • The data in the accountability metrics is useful and necessary for schools and districts to understand their performance • Understanding where you are as a school/district is the first step toward moving forward • Working smarter, not just harder

  9. Using the Top to Bottom Ranking • Three main components by subject: • Achievement • Improvement in achievement over time • The largest achievement gap between two subgroups calculated based on the top scoring 30% of students versus the bottom scoring 30% of students Each component tells schools something about their overall performance and can be used for diagnostics

  10. Who receives a ranking? Schools with 30+ full academic year (FAY) students over the last two years in at least two state-tested content areas; school must be OPEN at time of list generation Application  Some schools do not receive a ranking if they: • Have too few FAY students • Only have one year of data

  11. Tested Grades and Subjects • Reading and Mathematics: Grades 3-8 and 11 • In grades 3-8, testing every year allows us to figure out student performance level change (our current “growth” metric) in reading and math • Students can either significantly improve, improve, maintain, decline or significantly decline • Writing: Grades 4 & 7 • Science: Grades 5 & 8 • Social Studies: Grades 6 & 9

  12. What about Reconfigured schools? • A school must change by four or more grades in order to get a new code • Example: A K-2 building becoming a K-6 building. • New codes NOT granted when a school is reopened as a charter, for example • If not, the school retains the old code and continues to have data “point” at it from all students for whom that code is their feeder school • There is no “phase reset” like there was in AYP • If school population changed by 51%, could request a phase reset—still got AYP calculations, but sanctions delayed • Under Priority/Focus interventions, would simply have a customized intervention.

  13. What is a Z-Score? Quick Reference for Z-Scores

  14. Why do We Use Z Scores? • Z-scores are a standardized measure that helps you compare individual student (or school) data to the state average data (average scores across populations). • Z-scores allow us to “level the playing field” across grade levels and subjects • Each Z-score corresponds to a value in a normal distribution. A Z-Score will describe how much a value deviates from the mean. What do you need to know: Z-scores are used throughout the ranking to compare a school’s value on a certain component to the average value across all schools.

  15. What is a Z-Score? • Z-scores are centered around zero • Positive numbers mean the student or school is above the state average • Negative numbers mean the student or school is below the state average State Average …Worse than state average Better than state average…. 1 -3 -2 0 2 3 -1

  16. Z-Score Examples • Your school has a z-score of 1.5. You are better than the state average. Z-score of 1.5 State Average …Worse than state average Better than state average…. 1 -3 -2 0 2 3 -1

  17. Z-Score Examples • Your school has a z-score of .2. You are better than the state average, but not by a lot. Z-score of 0.2 Z-score of 1.5 State Average …Worse than state average Better than state average…. 1 -3 -2 0 2 3 -1

  18. Z-Score Examples • Your school has a z-score of -2.0. You are very far below state average. Z-score of -2.0 Z-score of 0.2 Z-score of 1.5 State Average …Worse than state average Better than state average…. 1 -3 -2 0 2 3 -1

  19. How Is the Top to Bottom Ranking Calculated Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Improvement Metric (Performance Level Change OR Four Year Improvement Slope) School Improvement Z-Score Content Index Z-score School Content Area Index 1/4 Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  20. How Is the Top to Bottom Ranking Calculated IMPORTANT PART RIGHT HERE!!! Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Improvement Metric (Performance Level Change OR Four Year Improvement Slope) School Improvement Z-Score Content Index Z-score School Content Area Index 1/4 Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  21. How do we get Standardized Scale Scores for Each Student? • Step #1: Take each student’s score on the test they took and compare that score to the statewide average for students who took that same test in the same grade and year • This creates a student-level z-score for each student in each content area • Compare • MEAP to MEAP • MEAP-Access to MEAP-Access • MME to MME • MI-Access • Participation to Participation • Supported Independence to Supported Independence • Functional Independence to Functional Independence

  22. What do we do with those standardized scores? • Step #2: Once each student has a z-score for each content area (based on the test they took), we take all of the students in a each school, and rank order the students within the school. • Z-scores will have come from different tests, and compare students to statewide average for that grade, test, and subject • But they can now be combined for the school • Step #3: Add up all z-scores and take the average. This is now the average standardized student scale score. • Step #4: Define the top and bottom 30% subgroups, based on that rank ordering.

  23. Average Z-score (average standardized student scale score): 0.28 (sum all z-scores, divide by 15)

  24. Top 30% Bottom 30%

  25. What is important to show Schools? Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Two-Year Average Performance Level Change Index School Performance Level Change Z-Score Content Index Z-score School Content Area Index 1/4 Step #1: Achievement How well did the school do in that subject? Positive number = better than average Near zero = average Negative number = worse than average Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  26. What is important to show Schools? Step #2: Improvement Is the school improving in that subject? Positive number = greater rate of improvement than average Near zero = average improvement Negative = slower rate of improvement than average; can also mean they are declining • For grade 3-8 reading and mathematics Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Improvement Score School Improvement Z-Score Content Index Z-score School Content Area Index 1/4 Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  27. What is important to show Schools? Step #3: Achievement Gap Is the gap in that subject between top 30% and bottom 30%: (positive number) = smaller gap than average (negative number) = larger gap than average (near zero) = average gap • For grade 3-8 reading and mathematics Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Improvement Value School Performance Level Change Z-Score Content Index Z-score School Content Area Index 1/4 Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  28. Once they have looked at each component, Discuss: • What’s the overall pattern? • Low achievement? • Declining achievement? • Large gaps? • Where are the actionable areas? • Which subjects need the most attention? • Is everyone doing poorly (small gap, low achievement) or are some students doing well and others falling behind (decent achievement, but large gap)

  29. What is important to show Schools? • For grade 3-8 reading and mathematics Two-Year Average Standardized Student Scale (Z) Score School Achievement Z-Score 1/2 Reward Schools (for improvement) Two-Year Average Performance Level Change Index School Performance Level Change Z-Score Content Index Z-score School Content Area Index 1/4 Focus Schools Two-Year Average Bottom 30% - Top 30% Z-Score Gap School Achievement Gap Z-Score 1/4

  30. An example from data: • My daughter’s elementary school • K-4 building • Go here to get this tool: • www.mi.gov/ttb

  31. Final Point • Accountability data does not tell ALL • “Correlation does not imply causation, but it does waggle its eyebrows suggestively and gesture furtively while mouthing ‘look over there’.” • Moving beyond the label only takes capacity at every level of the field

  32. Resources • www.mi.gov/ttb • Individual school lookup tool • Diagnostic Worksheet • www.mi.gov/priorityschools • www.mi.gov/focusschools • www.mi.gov/rewardschools

  33. Contact Information • Venessa A. Keesler • keeslerv@michigan.gov (personal email) • mde-accountability@michigan.gov • 877-560-8378, option 6

More Related