Measuring Up: Monitoring the Implementation and Development of a School Improvement Initiative Using a Program Theory Pe

1. Measuring Up: Monitoring the Implementation and Development of a School Improvement Initiative Using a Program Theory Perspective Shannon E. Coulter District and School Improvement, San Diego County Office of Education, San Diego, USA

2. Purpose • To examine • how professional learning communities (PLCs) operated in comparison to an example of a PLC model developed from criteria gathered from PLC research literature, and • if teachers’ connected their PLC beliefs and actions to changes in student engagement and achievement, and • whether relationships existed between teachers’ PLC beliefs and actions and standardized test scores.

3. Introduction • PLC Movement • PLCs emerged in the mid 1990s as an effort to improve teacher professional learning • Shifted professional development from “sit and get” type of workshops to job-embedded teacher work groups. • PLC attributes include shared values, reflective dialogue, collaboration, classroom observations, and analysing student achievement. • PLC theory of action suggests these short-term outcomes (i.e., PLC attributes) impact teacher learning, classroom practices and beliefs, and student achievement.

7. Study Questions • The main goal of this study was to explore a situation where a school district provided teachers with professional development training and support and determine the extent to which the training and support influenced teachers’ beliefs and practices and their students’ achievement. • The following questions guided this process: • To what extent are teachers’ PLC actions and beliefs congruent within and among schools and content areas? • To what extent are teachers’ PLC actions and beliefs related to their perceptions of PLC impacts? • To what extent are teachers’ PLC actions and beliefs related to student achievement?

8. Participants • 99 teachers (33 ELA, 32 math, 15 science, and 19 social science) randomly sampled from 241 district teachers • Teachers nested in 30 content area PLCs and 9 comprehensive high schools. • Content area teams composed of 4-7 teachers from the same content area and school; teams convened once a week for 45 minutes for “Collaboration Time.” • All 99 teachers trained during the 2003-2004 school years by national PLC consultants. • Consultants presented four full-day teacher leader workshops in areas of collaboration, analysing student data, creating assessments, and others.

9. School Characteristics

10. Measures • PCI Survey. Developed by Karen Seashore-Louis and Helen Marks measuring 5 PLC attributes (i.e., shared values, reflective dialogue, collaboration, classroom observations, and analysing student achievement). • PLC Outcomes Survey. Follow-up survey adapted from a Centre on Organization and Restructuring of Schools Survey (Newmann & Wehlage, 1995) measuring teachers’ perceptions of the impact PLCs have on teaching effectiveness, curriculum quality, student engagement, and equity. • Standardized achievement. Cross-sectional achievement data (California Standards Tests) from 2007-08 and 2008-09 school years. CST data averaged across teachers for two years to “smooth” data.

11. ELA PLC Actions and Beliefs

12. Math PLC Actions and Beliefs

13. Science PLC Actions and Beliefs

14. S_Science PLC Actions and Beliefs

15. PLC Actions and Beliefs by Perceptions of Impact

16. PLC Actions and Beliefs by Student Achievement

17. Discussion How does a school district’s professional development training and support influence teachers’ beliefs and practices and their students’ achievement? • District-level training and support led to vastly different understandings of PLCs (within and across content areas). • District-level training marginally influenced teachers’ beliefs that PLC activities led to important educational outcomes. • Teachers’ PLC perceptions and participation explains less than 2% of student achievement (standardized tests).

18. Discussion • To what extent are teachers’ PLC actions and beliefs congruent within and among schools? • PLC activities within a single school site varied dramatically; teachers appeared to focus on certain PLC attributes and exclude others. • For example, 60% of teachers specified no other teacher or administrator observed their classrooms throughout the year • PLC activities across the district also varied; when teachers within content areas returned to their sites PLC implementation appears to have been unregulated/unsupervised

19. Discussion • To what extent are teachers’ PLC actions and beliefs related to their perceptions of PLC impacts? • Teachers marginally linked positive educational outcomes like improved achievement and engagement to their professional community work; PLC actions explained about 35% of teachers’ perceptions of impact. • However, teachers used PLC time for low impact work. • For example, • 75% agreed they used PLC time mostly for curriculum coordination. • 95% indicated they spent less than half their collaboration time diagnosing student performance, and • 91% indicated they spent less than half their collaboration time discussing different teaching strategies with one-third reported never discussing teaching practices.

20. Discussion • To what extent are teachers’ PLC actions and beliefs related to student achievement? • PLC activities and beliefs appear largely unrelated to student achievement; for social studies teachers the relationship was negative. • PLCs experienced fidelity problems during implementation leading to an indeterminate influence on student achievement, especially in math, science, and English.

21. Recommendations • Make efforts to (a) ensure program materials and activities are delivered with fidelity and (b) participants implement PLC activities with fidelity. • Maintain a standardized set of expectations for all PLC trainings. Avoid differentiating the training for different content areas. • Regulate and supervise PLC implementation. • Monitor PLC activities for focus. Facilitate PLC discussions and conversations around high impact work such as diagnosing student performance and away from PLCs as “common planning sessions.” • Monitor teachers’ PLC perceptions. Facilitate conversations that link PLC work to changes in classroom work, beliefs, and student achievement.

22. Selected References • Elmore, R. F., Peterson, P., & McCarthey, S. (1996). Restructuring the classroom: Teaching, learning and school organization. San Francisco: Jossey-Bass. • Hord, S. M. (1997). Professional learning communities: Communities of continuous inquiry and improvement. Austin, TX: Southwest Educational Development Laboratory. • Lee, V. E., Smith, J. B. (1996). Collective responsibility for learning and its effects on gains in achievement for early secondary school students. American Journal of Education, 104, 103-147. • Louis, K. S., Marks, H. M. (1998). Does professional community affect the classroom? Teachers' work and student experiences in restructuring schools. American Journal of Education, 106, 532-575.

23. References • Newmann, F. M., & Wehlage, G. G. (1995). Successful school restructuring. Madison, Wis: Center on Organization and Restructuring of Schools. • Supovitz, J. A. (2002). Developing communities of instructional practices. Teachers College Record 104, 1591-1626.