Evaluating Growth in Tested and Non-Tested Subjects

1. Evaluating Growth in Tested and Non-Tested Subjects Two SLO pilots at the Denver Public Schools Michael Cohen, Denver Public Schools Elena Diaz-Bilello, National Center for the Improvement of Educational Assessment

2. Historical Context • First district in the country to implement SLOs (called Student Growth Objectives or SGOs) for high stakes (tied to pay) purposes • Originally implemented for Professional Compensation (ProComp) purposes in 16 pilot schools in 2001 • Deployed building support teams to provide ongoing PD and assistance to teachers AND administrators • Scaling up of SGOs resulted in the degradation of the process – compliance based and no longer deemed as meaningful by many teachers and administrators

3. Moving from SGOs to SLOs • Changed the name to signal a new beginning • 2013-2014 district-wide communications on new SLO process that will connect to district initiatives (CCSS implementation, cultivating formative assessment practice initiatives, and LEAP • Implemented research partnership with NCIEA and CU Boulder to pilot one version of the process in 3 schools • ARE staff piloted another version of the process in 12 schools

4. Two SLO Pilots Tier 2 - Implemented at 12 elementary, middle and high schools • Work with schools focused on keying in on priority standards as a way to help focus the SLO process for teachers Tier 3 • Implemented at 3 schools (1 elementary, 1 K-8, and 1 high school) in math and 2 schools (K-8 and high school) in visual arts • Work with schools focused on using learning trajectories or progressions as a framework for implementing the SLO process

5. Shift from SGOs and Evaluating Growth Tier 2

6. Shifting from SGOs The Tier II pilot model was designed in collaboration with NCIEA and CU Boulder, emphasizing the move toward: • A standards-based system for objective-setting • Use of a body of evidence/ongoing progress monitoring (instead of pre-/post-tests) • Differentiated targets • Team-based goals

7. Shifting from SGOs In implementation, SLOs are intended to: • Integrate naturally with other district initiatives: • Use of common short-cycle assessments • Team-based data inquiry • Increasing rigor through standards-based instruction • Be an authentic part of teaching and learning, not a compliance activity • Empower teachers to contribute their own measures of student growth for accountability purposes • Encourage teachers to aim high for student growth (moving away from a binary rating system to a 4-point scale that incentivizes ambitious goal-setting)

8. Shifting from SGOs Crucially, the support model must be intensive: • Coaching/mentoring in the context of data teams, not large-group training sessions • Frequent interaction with coaches from ARE (bi-weekly) • Involvement of school leaders

9. The Tier II SLO Model Ongoing Data-Driven Instruction Step 1: Create an Objective Statement Step 2: Determine Performance Criteria Step 3: Collect Baseline Data Step 4: Place Students in Baseline Groups Step 5: Set differentiated Targets Step 6: Plan and Collect Body of Evidence Step 7: Reflect and Evaluate Student Growth

10. The Objective Statement: Setting the Foundation • The Objective Statement is comprised of multiple standards and/or grade-level expectations and is a general description of what students will know and be able to do at the end of the course. • The selected standards should reflect the knowledge and/or skills that are critical for students’ success in the current course and future courses (including other content areas)

11. Some Sample Objective Statements Third Grade Literacy: All students will be able to write opinion pieces on a topic or text, supporting their point of view with reasons.  They will also be able to clearly introduce their topic, create an organizational structure that lists their reasons, and use linking words to connect their opinion and reasons. Finally, all students will provide a concluding statement in their opinion pieces.  Seventh Grade Mathematics: All students will be able to construct viable arguments and critique the reasoning of others verbally and in writing for real-world problems that analyze and use proportional relationships. Problems will include multi-step ratio/percent problems. High School Social Studies: All students will be able to create a written presentation using appropriate social studies tools and practices to analyze primary and secondary source documents to solve problems regarding continuity and change, cause and effect, complexity, unity and diversity over time.

12. Current SLO Evaluation and Scoring Model A three-step process: • Fall: Approval of Objective Statements, Performance Criteria, Baseline Data, Targets, and Plan for Body of Evidence • Mid-Year: Check-in to discuss progress of students and instructional interventions • EOY Conversation: • Discuss student outcomes in relation to the targets • Rate quality of measures using “Confidence in Data Rubric” • Use decision matrix, juxtaposing confidence in data against percentage of students meeting targets

13. Current SLO Evaluation and Scoring Model: Determining Student Growth • Teachers develop a rubric aligned to the Year-Long Objective, using 4-6 performance criteria, in the beginning of the school year • Teachers set targets for students, differentiated according to baseline data • Targets identify how much movement along the rubric is expected for each student • Rubric is used to track student growth over the course of the instructional timeframe • At EOY, teachers calculate the number of students who met their growth targets • Percentage of targets met in the student population is applied to a decision matrix for evaluating teacher’s SLO performance

14. Current SLO Evaluation and Scoring Model Confidence in Data Rubric

15. Current SLO Evaluation and Scoring Model Evaluation Matrix Four possible ratings: Exceeded, Met, Approached, Not Met

16. Lessons Learned: Successes • Increased knowledge of new standards and what proficiency looks like • Development of capacity in assessment and data analysis/use (becoming more critical of each data point and what can be inferred) • Intensive support from coaches • Empowerment of teachers as professionals

17. Lessons Learned: Challenges • Critical need for assessment and data support remains • Maintaining a routine of frequent coaching sessions in the context of busy school schedules and other—sometimes unrelated—PD initiatives • Designing a fair and feasible scoring model for rating SLOs (still unresolved) • Scaling up (from 14 to 140 schools) and providing sufficient support (e.g., finding the middle ground between the extremes of providing pre-fabricated objectives and having teachers start out with a blank slate) • Supporting teachers outside the Common Core (e.g., the arts) with collaboration opportunities, access to assessments, and equity

18. Shift from SGOs and Evaluating Growth Tier 3 (Visual Arts)

19. Distinction from Tier 2 • Tier 3: Implementing the use of Learning Trajectories as framework for SLOs at 2 school sites • Series of 5 full day workshops mixed with check-ins by both email and in person during the 2013-14 school year • Last workshop in May 2014

20. Across Grade (“Macro”) and Within Grade (“Micro”) Learning Trajectories MICRO LT (Your SLO) Level 4 Level 3.3 Level 3 Level 3.2 Level 2 Level 1 Level 3.1 MACRO LT

21. Creating a “Micro” Learning Trajectory What students should know and be able to do after instruction [SLO] The Process Pick the “upper anchor” first. What do you want all your students to know & be able to do? [Should be a level of a Macro LT] Set the “lower anchor” next. What do you expect them to already know & be able to do before you start teaching? [Should be a level of a Macro LT] Choices for 1 & 2 should be informed by the district’s evidence outcomes or from pre-existing Macro LT. Now add some hypotheses about how students get from lower to upper anchor. Base this on your experiences as a teacher and/or research in arts education (which is also based on experiences with students!) The “Messy Middle”: Partial Understanding, Useful Misconceptions What students should know and be able to do prior to instruction

22. Example of a draft micro LT developed for HS Level 1 Beginning Drawing Course

23. Student Work Products for Activities at Different Levels of Micro LT Level 1.2 – Self portrait in pencil Level 1.4 – Charcoal object drawing

24. Tier 3 Approach for Evaluating Student Growth • No predicted targets set by teachers (goes away from the approach taken by most places) • Used a value-table linked to the LT framework to establish values related to growth expectations • Differentiated point scheme still awards points to students who grow, but do not meet the objective • Teachers negotiate the points • We (NCIEA) defined the performance standards (i.e., what constitutes minimum, typical and high growth) with teacher agreement

25. Quantifying student growth G F E 1 Some level jumps may be more difficult than others. D 3 1 C B We can give each level-jump a score to give “credit” for more difficult jumps 2 A 1 1

26. Value-Table for Scoring Student Movement Performance on SLO by End of Course of Instruction

27. Definitions set in Tier 3 for minimal, acceptable, and aspirational growth • Minimal growth = Student moved 1 performance level up • Acceptable growth = Average score (use the median here) achieved in classroom reaches 2 or above (2 is the score associated with moving at least 2 levels up for 2 lowest groups; and for meeting objective for the highest group) • Aspirational growth = Student meets the SLO expectations

28. Pilot Rules: Translating outcomes into teacher ratings • More than half of students do not meet minimum growth. Rating : Low • At least half of all students meet minimum growth. Rating : Approaching • The median score achieve is equal to or higher than acceptable growth. Rating: Meets • All students had at least minimal growth and at least 80 percent of all students met aspirational growth (the objective). Rating: Exceeds

29. Tier 3 Visual Arts Challenges • LT concepts very difficult for the K-8 teachers • Immediately changed course and only focused on developing a within-unit trajectory for units and worked with them and 7 other teacher leaders in the Arts – Tier 3B • Continued on with original pilot design with high school • Difficult to construct an LT well when no district curriculum in the Arts and state standards in the arts deemed difficult to interpret and evaluate by arts teachers (consistent complaint) • More time needed to focus on cultivating assessment literacy skills and strengthening rubrics/tasks

30. Initial Successes • No longer focused on just making sure the SLO objective was met: more concerned about how much growth is each student making? • Clearer instructional targets being set and communicated to students • Definition of performance levels anchored to student work • Establishing clearer rationale for sequencing units of study