Student Learning Objectives : An Emerging State Framework for Design

1. Student Learning Objectives: An Emerging State Framework for Design Lessons from the Teacher and Leader Evaluation Multi-State Network January 11, 2013 DRAFT

2. Purpose of Today's Presentation As increasing numbers of states begin reforming their systems of educator evaluation to reflect a significant emphasis on student performance, there is growing demand to codify sound frameworks of Student Learning Objectives (SLOs), as an option for measuring teacher impact on growth beyond traditional growth or value-added models. • Several leading states are following the practice of early adopter districts and implementing SLOs as a measure of student growth. From that work, a framework and a concrete set of options have emerged. • The objective of today's presentation is to provide state teams that are designing and implementing new systems of educator evaluation with a framework for developing an SLO process, including an analysis of the opportunities and challenges of these measures as well as examples from early-adopter states. • Further, it lays out a vision for how SLOs might be used in the short-term and long-term to improve instructional practice, teacher effectiveness, and ultimately student achievement.

3. Options for Measuring Growth in NTGS Based on the work of several leading states and districts, there is a concrete set of options for measuring growth in non-tested grades and subjects (NTGS). These options, which are not mutually exclusive, are still emerging and will need to be refined and tightened over time to ensure quality. Options: • Extend assessment coverage to other subjects or grades through additional assessments developed or selected by the state or district. Examples include: Hillsboro ISD, Delaware, ACT, NWEA, end of course exams, etc. • Broaden attribution of student growth on current assessments to additional subject areas, such as attributing growth on math assessments to science, or more broadly to the grade or school-wide levels. Examples include: Delaware, Indiana, Tennessee, New York • Measure teacher progress against SLOs using district selected or teacher generated tests, end-of-course tests, or portfolio of student work. Examples include: DCPS, Austin ISD, New Haven, Indiana, Rhode Island, New York

4. Defining Student Learning Objectives "Student learning objectives" is a broad term used to describe a process to assess teacher impact on student performance that involves assessment selection and goal-setting at the district, school, or teacher level. • Notably, the term SLOs can imply a range of structures that can vary significantly in terms of the prescriptive nature of the process and the degree to which educators have the freedom to select individual assessments and targets.

5. An Overview of SLOs Recognizing that student learning is an important part of improving educator practice and a key underpinning of assessing teacher effectiveness, SLOs can play a critical role in new systems of educator evaluation in many states. • Promising practice suggests that multiple, meaningful measures of student growth are needed for strong evaluation systems, which can be particularly difficult for teachers who do not have data from state assessments and traditional growth models. • The use of SLOs in statewide systems of evaluation is a new and quickly evolving practice, drawing from the lessons-learned and promising practices of early-adopters. Although difficult to implement well, SLOs have the potential to continuously improve and advance over time to ensure stronger validity of the evaluation as well as greater impact on instructional quality. • Recognizing that present focus must be on establishing a basic system to meet immediate needs, states should not lose sight of the long-term vision in which SLOs can significantly improve teaching practice and support broader state college and career ready reforms, e.g. Common Core implementation.

6. Benefits and Challenges SLOs offer a number of benefits as evaluation measures, but also present significant challenges in implementation. As a result, states should be thoughtful about their use and treatment of SLOs in the near-term, recognizing that in the long-run, with added local capacity, SLOs could more deeply engage teachers in their evaluation and improve individual and collective teacher capacity. Benefits: • SLOs codify a process already embedded in effective teaching – using rich student data to set standards-based goals and monitoring progress towards those goals. This process represents a significant and meaningful shift in using high quality practice to inform and drive policy. • SLOs provide a way of measuring student growth/achievement that can be applied across a broad spectrum of subjects and grades, without the creation of new statewide assessments, which is both costly and time-consuming. • SLOs provide an opportunity to take a more robust approach in assessing teacher impact on student achievement, allowing for broad and specific goals reflecting the unique context of a school, classroom, and/or teacher. • SLOs allow teachers to take ownership of a portion of their evaluation, which can create buy-in and add authenticity to the process.

7. Benefits and Challenges, continued Benefits, continued: • Use of SLOs can drive professional learning for educators and provide a powerful link between evaluation and professional development to create a continuous improvement cycle. • SLOs can directly link the evaluation to the Common Core standards, even outside of reading and math, by targeting key teaching practices required for effective instruction of new, more rigorous curriculum Challenges: • Based on the experiences of early adopter states, most teachers and principals do not readily possess the necessary data and assessment literacy to develop strong assessments or SLOs. States must invest significant time and resources to develop educators' readiness in these areas. • In most states and districts, there is a lack of strong assessment coverage in many grades and subjects, but also a demonstrated lack of quality in many teacher-created assessments. States must provide clear training and guidance about the selection of quality assessments for use in setting SLOs. • SLOs can lower expectations for students if they are not set at a rigorous level. States will need to provide clear training and guidance as well as mechanisms for monitoring to ensure SLOs are set at an appropriate and consistent level of rigor for all teachers.

8. Early Adopter Districts Along with several other large school districts, District of Columbia Public Schools (DCPS) was an early adopter of SLO processes as part of its evaluation system. This district provides important lessons learned and insights to states as they develop systems for SLOs. DCPS: The district is in its fourth year of using an SLO process as part of teacher evaluation. Evaluation outcomes are also linked to compensation. Teachers in non-tested grades and subjects develop SLOs that count as 10% of their overall evaluation rating. The district provides a list of suggested assessments and targets for most grades and subjects, and all assessments must be rigorous, aligned to standards, and appropriate for the teacher's students. Targets can be based on either growth or mastery and are developed collaboratively by the teacher and the principal. When SLOs are set, a clearly defined rubric is developed to determine a teacher's SLO score based on student performance.

9. Emerging State Models Several leading states (DE, GA, IN, NY, RI) have recently begun piloting the use of SLOs as part of statewide systems of educator evaluation. These states provide an initial framework for designing systems of SLOs and offer early lessons learned as they continue to refine and improve SLO implementation. This presentation takes a closer look at two of those models. Indiana: SLOs are used for all teachers in the state default model, while districts developing their own model can choose to use them. In the default model, SLOs comprise 10%, 15%, or 20% of the overall evaluation, depending on the availability of growth model data. State guidance outlines criteria for assessment selection and specifies an order of preference that requires teachers use state assessments if available. Teachers must select a learning objective that applies to all students in a class and a targeted objective focused on low-performing students. SLOs are set collaboratively among teachers and approved by principals.

10. Emerging State Models, continued Rhode Island: All teachers must set between two and four SLOs. For teachers in non-tested grades and subjects, SLOs will comprise 51% of their evaluation. SLOs will be combined with value-added data for teachers in tested areas. Teacher and school-developed assessments must be reviewed on a quality rubric. Student learning targets must be aligned with school-level goals and be horizontally and vertically consistent across a school. Targets should be rigorous but attainable and if met, indicate that students have the essential knowledge and skills for the next level of instruction. Teachers and evaluators agree on SLOs at a beginning-of-year conference and meet mid-year to discuss progress and determine if any adjustments are needed.

11. Student Learning Objectives Framework Designing a student learning objective process requires states to address four core areas, as well as build capacity and continuously improve systems over time: • Selecting assessments • Setting targets • Reviewing results and scoring • Providing state supports In approaching this work, states must consider their objectives for the SLO process, what they will measure, how SLOs will be incorporated as part of an evaluation and growth system, and how to maximize the quality of the system over time.

12. Student Learning Objectives Framework • Selecting Assessments • Relevance • Stretch • Precision • Rigor • Attribution • Basis for Measurement • Timeliness • Adaptability • Setting Targets • Basis for Measurement • Type • Instructional Interval • Student Population • Breadth • Relevance • Consistency • Comparability • Timeliness • Rigor • Reviewing Results and Scoring • Roles and Responsibilities: Evidence and Teacher and Leader Input • Scoring Process: Assessment of Progress, Weighting, and Overall Scoring • Supporting Continuous Improvement: Scoring Security, External Validation, and Ongoing Development • Providing Systems and Supports • Analyzing Readiness • Capacity Building • Providing Strategic Communication • Monitoring • Ensuring Continuous Improvement • Soliciting Stakeholder Feedback

13. I. Selecting Assessments The first step in measuring student learning is to select appropriate assessments or sources of evidence.  Assessments may vary in format and design, but should meet minimum standards. Further, states must recognize that intensive professional development to support assessment literacy is essential in supporting this process. Key criteria include: • Relevance: Assessments should reflect knowledge and skills that are valuable to the student and school context, including alignment to state standards, the Common Core, and/or national standards for certain subjects (e.g. career and technical education). Additionally, assessments should align with goals for the student, school, or district. • Stretch: Assessments should measure student performance across a wide range of performance levels to ensure that they provide an accurate measure of student achievement/growth, including for those who perform significantly above or below grade level. • Precision: Assessments should be valid, accurate, and yield specific scores. While the psychometric standard of validity is not necessarily required, there should be evidence to suggest that the assessment is strongly aligned with course content and have enough variation in scale to yield scores that are meaningfully differentiated across performance levels.

14. I. Selecting Assessments, continued Key criteria, continued: • Relevance: Assessments should reflect knowledge and skills that are valuable to the student and school context, including alignment to state standards and the Common Core. • Rigor: Assessments should capture true mastery of skills aligned to college and career-ready expectations and higher order thinking skills, such as critical thinking, problem solving, collaboration, communication, etc. • Attribution: Assessments should yield data that can be assigned to individual teacher efforts (i.e. assessments must yield scores at the individual student level that can be linked to individual teachers based on state/district business rules). • Basis for Measurement: Assessments used to measure student growth should include a way to develop baseline data to benchmark progress (i.e. there should be some sort of pre-test and post-test available if a student learning objective is measuring growth). • Timeliness: Assessments should provide data in time for use in evaluations. (Some assessments such as AP tests that return results in July may not be able to be used for student learning objectives depending on the evaluation cycle and system design of the state/district). • Adaptability: Assessments should include the potential for accommodations and modifications for students with disabilities and English language learners.

15. I. Selecting Assessments, continued Key issues to consider: • Determine what types of assessments qualify, for example: • common assessments aligned to state standards, • assessments purchased or created by schools or districts, • classroom assessments, which may include teacher-created assessments, • project-based assessments, • performance-based assessments. • Establish and clearly articulate an order of preference for use of different assessment types. • Consider how the state will address the issue of assessment cost. • Consider how teachers, schools, and districts can work collaboratively to develop assessments. • Determine a process for approval of assessment selection. (Will the state play a supportive role by providing guidance and training to districts, or will it exercise stronger authority?)

16. DCPS Assessment Selection DCPS provides a list of suggested assessments for most grades and subjects, and outlines key criteria for assessment selection, including: • Alignment: Must be aligned to state standards and appropriate for teacher's class and students; exemplar/recommended SLOs developed by district for most subjects/grades. • Stretch: Not addressed, though district provides guidance on setting appropriate targets for students at various performance levels. • Precision: Not addressed. • Rigor: Must include multiple question types including higher levels of Bloom's taxonomy, real world application or problem-solving, and brief constructed response. • Attribution: Required. • Basis for Measurement: Performance tasks must have a pre-test. component. • Timely: Required. • Adaptable: Required.

17. Indiana Assessment Selection State guidance for the default model specifies that teachers use state assessments if available, or, if not, common school or district assessments, or individual classroom assessments, if there are no other teachers in the content area. The state provides a pre-approved list of assessments, and requires evaluators approve any additional assessments. State guidance outlines key criteria for assessment selection: • Alignment: Must cover most key content standards; the state requires completion of the Alignment and Coverage Check Chart. • Stretch: Must assess pre-requisite objectives from prior years and objectives from the next year's course. • Precision: Must adhere to state established order of preference. • Rigor: Required – the state provides an Assessment Rigor Analysis chart. • Attribution: Required. • Basis for Measurement: Required in target-setting guidance. • Timely: Must provide score reporting in time for evaluation. • Adaptable: State provides guidance for local decision-making.

18. Rhode Island Assessment Selection Rhode Island requires all teachers to set between two and four SLOs. Teacher and school-developed assessments must be reviewed on a quality rubric reflecting the following criteria: • Alignment: Must be aligned to state standards (and show evidence of this alignment); must be paced with curriculum; common assessments must be used if available. • Stretch: Must include varied response types (e.g. multiple choice, constructed response, etc.) to assess complexity of content. • Precision: To ensure validity, must include at least three versions of a test if given multiple times throughout year. • Rigor: Must include some evidence of higher-order thinking. • Attribution: Required. • Basis for Measurement: Required in target-setting guidance. • Timely: Must provide score reporting in time for evaluation. • Adaptable: Must make accommodations available to ensure equitable testing.

19. II. Setting Learning Objective Targets Once a high quality assessment has been selected, it is essential to develop strong student learning objectives that set ambitious but achievable expectations for students and that provide teachers with clear instructional goals. SLO processes should clearly articulate a process for setting targets that identifies who is responsible for setting and approving targets and what steps are in place to ensure quality. States should also ensure training and professional development Key considerations include: • Basis for Measurement: Targets should distinguish different levels of performance mapped to a clear rubric (i.e. targets must set specific student performance expectations with clear thresholds). • Type: States/districts must determine if targets can be based on student achievement (based on mastery of course content) and/or student growth (improvement in performance using baseline data). • Instructional Interval: Targets should specify the duration of instruction during which performance will be measured. • Student Population: States/districts should determine if targets should be set for all students, particular subgroups of students, or both.

20. II. Setting Learning Objective Targets, continued Key considerations, continued: • Breadth: States/districts should determine if targets should address all content standards for the course, or if a subset is permissible. (How much of course content standards must be covered to be sufficiently meaningful while ensuring targets are not overly broad/vague?) • Relevance: Targets should be meaningful to school and student context. • Rigor: Targets must be set at a level that corresponds with ambitious but achievable outcomes. • Consistency: States/districts/schools may want to require or recommend that teachers of the same grade or subject have the same or similar targets. (Vertical or horizontal alignment of targets can increase quality and provide greater cohesion.) • Comparability: States/districts should consider strategies to make student learning targets comparable to other measures of student growth. • Timeliness: Targets must be set by a particular point in the school year and measured at a particular point(s) in the year (i.e. approval must occur within a certain timeframe and final evaluations must reflect on data outcomes by a certain deadline).

21. II. Setting Learning Objective Targets, continued A clear process for setting and approving SLOs must be developed that articulates requirements and areas of flexibility. • What is the process for setting objective targets? • Who sets them? • Who approves them, and what's the process for ensuring quality? • How are teachers involved? • How clearly articulated are different levels of performance? (Will the state provide standards for each level? Exemplars?) • What is the process for tracking teacher progress toward targets and adjusting instruction as necessary? • Is there a process for revising learning objectives mid-year if it becomes clear they can be improved? In what circumstances is this appropriate?

22. District of Columbia Targets DCPS requires principals to review and approve all SLOs to ensure that they meet quality criteria and also conducts a central office review of feasibility. The district provides sample SLOs for most grades/subjects. • Measurable: Targets can be mastery or growth and must be anchored in baseline data. • Breadth: Targets must cover all students that a teacher instructs and cover the majority of content for a given course. • Rigor: Targets must represent an ambitious but achievable level of performance and should set a goal of 1.25 years of academic growth. • Meaningful: Targets should be based on the specific needs of the students a teacher instructs. • Consistency: Sample targets are provided for most subjects and grades. ELL targets must be based on state AMOs. Standard rubrics are available for some subjects. • Timing: Targets must be set at initial conferences (by October) with data available in time to inform a final evaluation (in June). DCPS reviews each target for feasibility (not quality).

23. Indiana Targets Indiana provides teachers and evaluators with a checklist of key steps for determining targets. It also provides exemplar and non-exemplar SLOs. • Measurable: Targets must be measurable and based on previous performance. Targets focus on student mastery. • Breadth: Teachers must select a learning objective that applies to all students in a class and a targeted objective focused on low-performing students. State guidance outlines three levels of student preparedness on which to base targets. Targets should cover all major standards and skills. • Rigor: Relies upon the judgment of the teacher and evaluator. • Meaningful: In setting targeted objectives, teachers must consider those students that enter the course inadequately prepared and may choose to focus on a few key standards or all standards. • Consistency: Teachers in subjects/grades covered by an approved assessment must adhere to state established content mastery standards. The state provides guidance on mastery using other assessments. • Timing: Targets must be set at initial conferences (by October), with a mid-course check-in to discuss progress (in January -February).

24. Rhode Island Targets Rhode Island provides teachers and principals with a checklist of criteria to evaluate targets. The state also provides exemplar and non-exemplar SLOs. • Measurable: Targets must be measurable and based on baseline/historical data or previous classes' performance. Targets can be mastery or growth. • Breadth: Targets must cover all students, but can include multiple tiered groups for subpopulations of students. Targets should cover all major standards and skills. • Rigor: Targets must correspond with at least one year of learning and attainment must suggest that students are on track in that subject/course. • Meaningful: Targets must include a rationale for the selection of the target based on student needs. • Consistency: Teachers across subjects/grades within a school should have the same targets. Targets should be vertically aligned with school and district priorities. • Timing: Targets must be set at initial conferences (by October) and then revised (in January) if there are major changes in class composition or other factors that would necessitate a change.

25. III. Reviewing Results and Scoring Following completion of the SLO process, evaluators and teachers must review SLO results and score the teacher's performance. This process should be articulated when targets are set. Depending on the level of local control, processes may be established by the state or district, and may even be subject to collective bargaining. Key considerations include: • Evidence: Articulate the responsibilities of teachers and evaluators in gathering and evaluating evidence of student performance and what amount of evidence is sufficient. • Teacher and Leader Input: Determine what level of professional judgment is to be used in making judgments and what the process is for teacher feedback and input. • Assessment of Progress: Articulate a process for making a judgment about teacher performance on student learning objectives, including who makes the judgments, by when, and whether other evidence or information should be considered in making a scoring determination. • Weighting: If teachers set multiple SLOs, determine how these should be weighted. • Overall Scoring: Determine how judgments and evidence from SLOs are incorporated into the larger summative evaluation.

26. III. Reviewing Results and Scoring, continued Key considerations include, continued: • Scoring Security: States/districts may want to establish systems to ensure greater scoring security. • External Validation: States should consider whether to develop a process to validate ratings, such as through correlational analysis with other measures or spot-checking/auditing of results at the school or district level. • Ongoing Development: States/districts must articulate how the SLO process is linked to feedback for professional development opportunities based on results. • Stakeholder Feedback: States/districts should develop systems to gather stakeholder feedback throughout the SLO design and implementation process to inform continuous improvement.

27. IV. State Systems and Supports While the role of the state may vary considerably depending on the context and degree of local control, there are a set of core functions and supports that a state should anticipate providing in order to implement a successful SLO process, including: • Analyzing Readiness: States should be prepared to analyze district readiness in terms of the level of assessment and data literacy among teachers and principals, and use this information to shape and differentiate training and professional development. • Capacity Building: States should anticipate providing or facilitating high-quality training on assessment selection, assessment creation, use of data, and goal-setting. • Providing Strategic Communication: States should develop a strategic communications plan for educator evaluation efforts of which SLOs are an important component. • Monitoring: States should develop policies and protocols to monitor and assess the quality of SLO implementation to ensure SLO outcomes align with other evaluation data, are not disproportionally impacting certain teachers, and are being implemented with rigor and fidelity. • Ensuring Continuous Improvement: States should develop a continuous improvement process to evaluate and refine the SLO process over time. This process should address perceived deficiencies as well as look for ways to enrich the SLO process.

28. DCPS Reviewing Results and Scoring DCPS principals judge evidence of student learning through SLO outcomes and assign a rating aligned with the rubric established at the initial conference. • Evidence: Teachers must provide evidence that student performance has met targets at their year-end conferences with evaluators. • Assessment of Progress: DCPS provides a general rubric with suggested criteria for each rating level. Principals and teachers agree on a rubric at the beginning of the year. At the end of the year, principals validate student scores and assess student progress against this rubric. • Teacher and Leader Input: Teachers input is given when SLOs are set (at an initial conference). • External Validation: Not addressed. • Overall Scoring: SLOs are rated according to a 4-level rating scale that is specified at the initial conference. • Ongoing Development: Principals and teacher discuss student learning outcomes at the end of the year to identify a final rating and determine professional growth strategies. This supplements ongoing conversations with principals and master educators on observational feedback.

29. Indiana Reviewing Results and Scoring Indiana evaluators judge evidence of student learning through SLO outcomes and, following discussion with the teacher, assign a rating based on professional judgment. • Evidence: Teachers must provide evidence of SLOs 48 hours prior to their summative conference with evaluators. Evidence of class objectives should only include student scores on the assessment, while targeted objectives may also require additional evidence, such as class work, student projects, etc. • Assessment of Progress: Evaluators use professional judgment to evaluate student learning in comparison to established targets and then determine a rating, based on four performance levels, for each objective. • Teacher and Leader Input: SLOs are set collaboratively among teachers and approved by evaluators. • External Validation: Not addressed. • Overall Scoring: All teachers receive SLO ratings on a 4-point scale. A weighted average is used to determine the final SLO score. (Class and targeted objectives count equally.) • Ongoing Development: Evaluators and teacher discuss student attainment of learning objectives, and the evaluator establishes ways to support the teacher.

30. Rhode Island Reviewing Results and Scoring Rhode Island principals judge evidence of student learning through SLO outcomes and assign a rating based on professional judgment. • Evidence: Teachers must provide evidence of SLOs prior to their final conferences with principals. • Assessment of Progress: Evaluators use professional judgment to evaluate student learning in comparison to established targets and then rate performance against all targets on a five-point scale. • Teacher and Leader Input: Teacher input is given when SLOs are set (at an initial conference). • External Validation: Not addressed. • Overall Scoring: All teachers receive SLO ratings on a 5-point scale based on professional judgment of their performance across 2-4 SLOs. • Ongoing Development: Evaluators and teacher discuss student attainment of learning objectives, what strategies were effective and ineffective, and what the teacher learned about curriculum, assessments, and instruction through the SLO process.

31. A Key Tension: Standardizations v. Individualization In designing a process for SLOs, states must balance the desire for consistency and comparability of rigor with the desire to promote teacher buy-in, engagement, and professional learning by allowing educators to select their assessments and targets. • Georgia designed an SLO process that requires districts to select assessments and targets for each grade and subject that are then implemented by all teachers across the district. This ensures that all teachers are setting rigorous targets that are comparable, but does not allow for significant engagement, use of baseline data, or personalization. • DCPS allows teachers to set their own targets and select assessments (subject to principal approval). While DCPS works to boost comparability and rigor by providing extensive training and SLO models for each grade and subject, there is a significant amount of variability in the system, and the district limits the weight of SLOs to 15% of the evaluation system. • Some states, like Delaware, have sought balance on this spectrum by employing SLO rubrics, pre-approved lists of assessments, and rich guidance and state supports as a means to ensure a sufficient level of quality and consistency while promoting as much flexibility and authenticity as possible for teachers.

32. The Potential for SLOs in the Long-Term Over time, SLOs can evolve through continuous improvement and thoughtful phase-in to allow for greater teacher engagement and more specific development of individual and collective capacity. • In the early years, while educator assessment and data literacy is limited, states may want to design SLO processes that are grounded in specific criteria with a narrower range of options for assessments, targets, and scoring. • However, as educators gain expertise in this process and as districts demonstrate increased capacity, states could consider loosening requirements to allow for more personalization of the system to promote greater educator ownership of the evaluation and to more fully incorporate professional development goals.

33. Questions? Questions? Robin Gelinas Berkley, EducationCounsel: robin.berkley@educaitoncounsel.com Margie Yeager, EducationCounsel: margery.yeager@educationcounsel.com