Research Indicators for Sustaining and Institutionalizing Change

1. Research Indicators for Sustaining and Institutionalizing Change CaMSP Network Meeting April 4 & 5, 2011 Sacramento, CA Mikala L. Rahn, PhD Public Works, Inc.

2. Session Overview • CaMSP Evaluation Overview--What we can learn moving forward • Emerging themes from MSP meeting, morning presentation, and panels on Institutionalization and Sustainability • Time for discussion and next steps after the Network meeting

3. Statewide Evaluation Research Questions • How have the Partnerships ensured that all students have access to, are prepared for, and are encouraged to participate and succeed in challenging and advanced mathematics and science courses? • How have the Partnerships enhanced the quality of the mathematics and science teacher workforce? • What evidence-based outcomes from the Partnerships contribute to our understanding of how students effectively learn mathematics and science?

4. Key Features of CaMSP Features of the programs reviewed based on the legislation: • Partnership driven • Teacher quality • Challenging courses and curricula • Evidence-based design and outcomes • Institutional change and sustainability

5. Critical Attributes of Professional Development:  Content focus  Active learning  Coherence  Duration  Collective participation Increased teacher knowledge and skills; change in attitudes and beliefs Improved student learning Change in Instruction L. Desimone Conceptual Framework FIGURE 1: Desimone Conceptual Framework for studying the effects of professional development on teachers and students

6. Content Focus Active Learning Coherence Duration Collective Participation IHE’s bring content lens Strengthened partnerships-stronger models Standards/textbooks/ assessments embedded 84 hours/3 years per teacher Collaboration embedded in classroom follow-up What’s the Match?

7. CaMSP in California • Nine cohorts funded so far; science grades 3 through 8 or mathematics grades 3 through Algebra I. • 59 partnerships included in the 07-08 outcome study; 88 partnerships in the 08-09 and 09-10 outcome study • PD models currently incorporate significant hours of training for three funding cycles--less attrition as implementation requirements tightened in California • Same cohort of teachers--we have data to measure based on consistent dosage and rules for implementation.

8. CaMSP Outcome Study • 07-08 Outcome Study completed (Report available) • 08-09 and 09-10 Outcome Study will be combined and reported Spring 2011 • Large studies: • 07-08: 284,538 treatment and comparison students/1,581 treatment teachers • 08-09 and 09-10: 165,209 treatment and comparison students/1,594 treatment teachers

9. CaMSP Evaluation and Outcome Study Results • A positive and statistically significant—though small—effect on overall mathematics CST scores in 2008 and 2009. Also showed a significant, and more dramatic, effect on Algebra I test scores in 2008 and 2009. However, not in 2010. • Science partnerships appeared to have a slight, though positive, impact on science learning in California in 2008, 2009, and 2010. This is mostly attributable to 8th grade performance. Limited to the 5th and 8th grades where the science CST is administered.

10. CaMSP Evaluation and Outcome Study Results (cont.) • A more in-depth analysis in the 2008-09 and 2009-10 study of partnerships by dosage/funding cycles completed reveal the diversity of mathematics and science achievement among partnerships. Some partnerships showed marked improvement, while others were little different from the comparison groups.

11. What are we learning from CaMSP? • There is a lot of variation in implementation despite rules--ability to be innovative continues to exist yet there are challenges to maintain cohort • LEA’s are getting used to their role in building partnerships with individuals from IHE’s and building capacity for managing high quality PD, especially in medium size districts; however, rural districts are particularly challenged by participation rules

12. What are we learning from CaMSP? • Individuals from IHE’s are tailoring their training and challenging teachers with content knowledge in ways that meet teacher needs often building on experience in Subject matter projects • Subject matter projects, CPEC grants and other IHE-led professional development exist in parallel with CaMSP but not necessarily integrated

13. What are we learning from CaMSP? • Structure and personnel in the classroom follow-up piece is essential to successful implementation and has improved; adaptation to teacher needs is important but fidelity to follow-up model supports institutionalization • Teacher leaders are emerging in many partnerships and being incorporated in training as facilitators and in district curriculum decision-making committees

14. What are we learning from CaMSP? • Lack of involvement of site administrators in planning and PD continues as an area of concern • Visibility of and continued focus on local evaluation is beginning to lead to higher quality instrumentation and documentation but there is little sharing of what we are learning

15. Emerging Themes • General panel on Institutionalization to address changes in policy and practice among the partners and what lessons the partnership has taken from the MSP project and infused into the district's culture. • General panel on Sustainability to address funding the effort once the grant sunsets. • ESEA Reauthorization and preparing to respond to the direction for MSP program

16. Discussion & Wrap-up • What has been most successful in moving our partnership forward? • Institutionalization of the model • Integration in district or IHE policy and practice • What change can we document as a result of this effort? • How do these efforts contribute to a competitive position for California in mathematics and science education?