Challenges in Quantifying and Qualifying Student Growth & Achievement

1. Challenges in Quantifying and Qualifying Student Growth & Achievement MASPA Conference, November 29, 2012 Presented by: Nick Kalakailo, Director of Student Growth & Accountability Novi Community School District

2. On my mind… • HB 6004 and SB1358 - EAA • HB 5923 – Selective Enrollment Schools • SB 620 – Parent Trigger • ALEC • These issues will impact your ability to comply with RSC 1249 • Available student growth and assessment data for (3) three years? Use most recent (3) three years. • No student growth and assessment data for (3) three years? Use what’s available.

3. Growth versus Achievement? Some learn to race Everyone should learn to swim • If there is only one “fastest swimmer” in the world, are the rest of us failures, or are the rest of us growing at a “slow rate?” Neither? • Can we all be the fastest? • When we are all the “fastest swimmers” will we have success? • Are we lacking achievement when we have no gold medals? Everyone learns about water safety

4. What kinds of language? • growth • progress • achievement • pre and post test • value added • evaluation • college and career ready • proficient • not proficient • effective • ineffective • content standards • professional standards • UICs • PICs • SCEDs • MEAP • MME/ACT • BAA • TSDL • CEPI • REP • MISchool Data • FOIA

6. Campbell’s Law and the Dangers of Quantification • Over thirty years ago, Donald Campbell pointed out that overreliance on imperfect quantitative measures of performance can do real harm. (Campbell, 1975) He formulated what is now known as Campbell’s Law: • “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.”

7. What kinds of language? • growth is about the individual • progress relates to gaps between current achievement and desired achievement • achievement relates to a level of performance at a given time (e.g. expected or grade level performance, performance level) • value-added (doesn’t work)

8. Value-Added? • In 1980s, William Sanders agricultural adjunct professor at University of Tennessee • Used a statistical method to track cattle breeding to predict the “passing on” of particular genetic traits • He said this could method be used to assess teachers ability to impact test scores or student growth • https://www.youtube.com/watch?v=18R2af2CKk8

9. From L. Darling-Hammond • “First, we found that value-added models of teacher effectiveness are highly unstable. • Second, teachers' value-added ratings are significantly affected by differences in the students who are assigned to them.” • November 5, 2012 Los Angeles Times.

10. From L. Darling-Hammond • We found that a teacher receives a higher value-added score when he is teaching students who are already higher-achieving, more affluent and more versed in English than when he is assigned large numbers of new English learners and students with fewer educational advantages. • Finally, value-added ratings cannot disentangle the many home, school and student factors that influence learning gains. • November 5, 2012 Los Angeles Times.

11. From L. Darling-Hammond • National Research Council, concluded: "VAM estimates of teacher effectiveness … should not be used to make operational decisions because such estimates are far too unstable to be considered fair or reliable.” • It illustrates that, as in other professions, good evaluation starts with rigorous, ongoing assessment by experts who review teachers' instruction based on professional standards. • “Evaluators look at classroom practice, plus evidence of student learning from a range of classroom work that includes (but is not limited to) school or district tests that directly connect with the curriculum and students.” • November 5, 2012 Los Angeles Times.

12. Which is Best (teachers)? • 1= measurement is the sole purpose of educator evaluation • 2= measurement and development serve a dual purpose, but measurement is dominant • 3= half and half – purpose of educator evaluation should be half development and half measurement • 4= development and measurement serve a dual purpose, but development is dominant • 5= developmentis the sole purpose of educator evaluation • Marzano - Ed. Leadership, Nov. 2012

13. Which is Best(students)? • 1= measurement is the sole purpose of student assessment (OF Learning) • 2= measurement and development serve a dual purpose, but measurement is dominant • 3= half and half – purpose of student assessment should be half development and half measurement (OF and FOR Learning) • 4= development and measurement serve a dual purpose, but development is dominant (OF, FOR, AS) • 5= developmentis the sole purpose of student assessment (FOR and AS Learning)

14. Measuring or Developing? • Providing clear lesson objectives • Understanding students’ background and comfort with the material • Using more than one delivery mechanism • Providing multiple examples • Providing appropriate nonexamples • Maintaining an effective pace • Providing students feedback • Timely use of guided practice • Explaining important concepts clearly • Active engagement throughout the lesson • RATE (Strong, M. via Marzano) • 4 Domains • Classroom Behaviors & Routines • 41 Indicators • Planning and Preparing • 8 Indicators • Reflecting on Teaching • 5 Indicators • Professionalism and Collegiality • 6 Indicators Measuring Teachers Developing Teachers • Marzano - Ed. Leadership, Nov. 2012

15. Adult Reading 5th Grade Beginning Literacy Assumptions of ‘Graded’ Schools • Curriculum scale • Skills taught may begin with associating sounds with letters in K-1, to adult levels in high school • Typically, we feel restricted to “covering” specific curricula related to adopted grade level standards Does this design meet the needs of all students?

16. Instructional Level vs. Mastery • The NWEA test provides the instructional level of the student. • It provides a road map for students toward achieving mastery. • It is not a test for determining mastery of skills.

17. Features of MAP Test • Challenging, appropriate and dynamically developed for every student • Accurate data for students across the scale • Untimed • Purpose is internal accountability • Measures growth in student achievement • Immediate results • In Novi, we assess 2 times a year

18. Sinking or Floating? • Still have a sinking feeling? • Sinking but seeking help? • Dog paddling in desperation? • Floating? • Swimming?

19. So what do we need to better communicate growth and achievement? What are the multiple data points that we intend to use and construct to better inform students and parents about progress and achievement?

20. Emerging Dashboard 2011-2012

21. Growing Dashboard 2012-2013

22. Growing Dashboard 2013-2014

23. Balanced Dashboard 2014-2015 SBG

24. More language…. • attribution • Who teaches what and whom? • congruence • Who assesses, who is assessed, what assessments, aligned to what? Does evidence reflect values? • assessment literacy • How do teachers use the data? Regardless of high or low results, a teacher’s ability to discern the meaning of the data and make use of it should be the measure of effectiveness. • accountability • How do we comply without doing harm to kids or adults? • stability

25. Thank You (and Questions?) • Nick Kalakailo • Director of Student Growth and Accountability • Novi Community School District • nkalakailo@novi.k12.mi.us

26. Michigan’s Teacher Evaluation Requirements • Michigan requires that teachers be rated as • Ineffective • Minimally Effective • Effective • Highly Effective • Remember: Beginning in 2013-14 student growth data will also be included in the evaluation system.

27. Translating the iObservation Scale into Michigan’s Ratings • Ineffective • Rated consistently on iObservation as “beginning” or below • Minimally Effective • Rated consistently on iObservation as “developing” with some “beginning” • Effective • Rated consistently on iObservation as “applying” with some “developing” • Highly Effective • Rated consistently on iObservation as “applying” with some “innovating”