How do you know if it’s working? Using feedback loops to assess progress

1. How do you know if it’s working? Using feedback loops to assess progress August 25, 2011

2. Today’s Agenda 2:30 – 2:35 Introduction Time Session 2:35 – 2:45 Feedback loops 2:45 – 2:55 2:55 – 3:20 3:20 – 3:30 Example: Louisiana Exercise: creating feedback loops Implications for Common Core implementation

3. Today’s Agenda 2:30 – 2:35 Introduction Time Session 2:35 – 2:45 Feedback loops 2:45 – 2:55 2:55 – 3:20 3:20 – 3:30 Example: Louisiana Exercise: creating feedback loops Implications for Common Core implementation

4. Delivery chains help to define the “intent” of a given strategy Sample delivery chain: professional development What percent of teachers will change their teaching practice as a result? PLC Facilitators Chief 1 Curriculum/ instruction team Teachers 65,000 Teachers 65,000 150 400 Principals 1 Principals 65,000 Teachers 25 PD instructors 25 Approved providers 700 15 Regional committees Curriculum directors 400 Region/ County State District School Classroom 10% (large districts) Train by 9/2013 Manage Train by 5/2012 15% (small schools) Train by 9/2011 Train by 12/2011 Give incentives to choose “approved” providers by 9/2011 Approve by 9/2011 Contract by 6/2012 Manage 15% (large schools) Train by 9/2013

5. We analyze delivery chains in order to identify weaknesses and address them Sample delivery chain: professional development 400 Chief 1 Curriculum/ instruction team 1 65,000 Teachers 65,000 PLC Facilitators Regional committees Principals Teachers Curriculum directors 150 Approved providers 25 PD instructors 25 Teachers 65,000 Principals 700 15 400 1 2 3 2 3 1 Region/ County What are the potential weaknesses? State District School Classroom Historically difficult relationship and loose authority structure – curriculum directors not likely to listen to regional centers Large number of principals at large high schools who are skeptical of state involvement; sheer communication will be a challenge Lobbying efforts by providers might prevent creation of “approved” list

6. There are four main types of measures that you can use Description Sample measures (PD) Processes and milestones Extent to which the processes and milestones of the strategy occur as they were intended to do • Number (and identity) of teachers and/or principals who have received aligned PD Project management User Satisfaction Extent to which front line and/or end users have a positive view of the strategy’s impact on their work • Number (and identity) of teachers and/or principals expressing satisfaction with aligned PD Change in front line knowledge or practice Extent to which those near the “end” of the delivery chain change their practices as intended by the strategy • Improved knowledge of Common Core State Standards (as measured by an assessment tool) for those who have received aligned PD vs. those who have not • Observations of practice for a sample of teachers that have and have not received aligned PD Quality of implementation Impact on student outcomes Extent to which students who are exposed to changed practices demonstrate better results than those who are not • Formative or summative assessment data, comparing teachers who have received aligned PD with those who have not

7. The delivery chain will help you to decide which measures to focus on Sample delivery chain with measures identified: professional development 400 Chief 1 Curriculum/ instruction team 1 Teachers 65,000 Teachers 65,000 Teachers Principals PLC Facilitators Curriculum directors 700 Principals 400 25 PD instructors 65,000 Approved providers 15 Regional committees 150 25 3 5 5 1 2 1 3 5 5 4 4 4 2 2 6 4 Region/ County What are the metrics we will use? State District School Classroom Number of teachers satisfied with training Number (and identity) of teachers trained by school officials Number of teachers trained whose observed practices are changing Difference in formative assessment gains for students with trained teachers vs. untrained Number of districts undergoing training Number (and identity) of teachers trained by approved providers

8. Once you have prioritized your metrics, you need to ensure that collection mechanisms are in place for each Measures from delivery chain (professional development) 5 4 6 2 3 1 Potential ways to collect data Number of districts undergoing training • Fold into district monthly reporting Number (and identity) of teachers trained by school officials • Fold into district monthly reporting • Collect directly from schools Number (and identity) of teachers trained by approved providers • Include a requirement for reporting on this information in vendor contracts Number of teachers satisfied with training • Add relevant questions to existing school climate survey Number of teachers trained whose observed practices are changing • Extrapolate from sample focus groups of principals, as well as existing principal advisory group Difference in formative assessment gains for students with trained teachers vs. untrained • Formative assessment data combined with survey self-reporting of adoption in classrooms

9. Today’s Agenda 2:30 – 2:35 Introduction Time Session 2:35 – 2:45 Feedback loops 2:45 – 2:55 2:55 – 3:20 3:20 – 3:30 Example: Louisiana Exercise: creating feedback loops Implications for Common Core implementation

10. Louisiana: creating conditions for actionable implementation information WHY do we want the information? WHAT will we do with it? WILL our informational needs be met? Informational NEED Sample measures Who Change in front line knowledge or practice • Do teachers know the content when they leave the PD? Do they maintain this knowledge? • Pre- and post- assessment of knowledge • Computer • Are teachers able to change howthey instruct and what they instruct? • Observational Rubric • with defined levels of implementation • Anyone – even students Impact on student outcomes • Which students should this PD impact? • Specific teacher data to link to students • PD providers to State DOE • Do students acquire the skills they need? • Formative assessment • Teachers to State DOE

11. Today’s Agenda 2:30 – 2:35 Introduction Time Session 2:35 – 2:45 Feedback loops 2:45 – 2:55 2:55 – 3:20 3:20 – 3:30 Example: Louisiana Exercise: creating feedback loops Implications for Common Core implementation

12. Exercise: Critique a delivery chain What How Materials Time • Look at sample delivery chain for instructional materials and fill out the template: • What kinds of measures would you want to track? • Are there any measures you would want to emphasize, in light of the identified weaknesses in the chain? • Are your measures balanced across the four types? • What are some practical and workable ways to get the data? • Discussion in breakout rooms • Whiteboard templates • 20

13. Delivery chain: instructional materials 1 Principals 15 Chief Curriculum/ instruction team Website Regional centers Curriculum directors Curriculum committees 150 Teachers 150 Teachers Teachers 400 1500 65,000 Coaches Teachers 65,000 1 65,000 65,000 1 Region/ County Intent: teachers will improve their practices State District School Classroom 10% (small districts) Manage Train and distribute by 9/2013 Train and distribute by 9/2012 15% (large districts) Train and distribute by 9/2011 Train and distribute by 5/2012 Consult 15% (large districts) Train and Distribute by 9/2013 Post by 9/2011 5% (mix) Market by 9/2012

14. Analysis of weaknesses in delivery chain Teachers Chief Curriculum/ instruction team Website Regional centers Curriculum committees Principals Teachers 1 Teachers Coaches Curriculum directors 1 1 65,000 65,000 65,000 65,000 1500 400 150 150 15 Teachers 2 3 1 3 1 2 Region/ County What are the potential weaknesses? State District School Classroom Historically difficult relationship and loose authority structure – curriculum directors not likely to listen to regional centers In smaller districts, principals may not have capacity to train their teachers A large proportion of teachers are not accustomed to using any state website to receive curricular resources

15. Feedback loops Measures Potential ways to collect data Processes and milestones User satisfaction Change in front line knowledge or practice Impact on student outcomes

16. Today’s Agenda 2:30 – 2:35 Introduction Time Session 2:35 – 2:45 Feedback loops 2:45 – 2:55 2:55 – 3:20 3:20 – 3:30 Example: Louisiana Exercise: creating feedback loops Implications for Common Core implementation

17. Discussion questions • To what extent does our current implementation plan for Common Core incorporate feedback loops? • How can we build feedback loops in areas where we do not currently have them? • Is there anything unclear about how to create feedback loops that we can help you with?