Evaluation of Xtreme Reading Implementation

1. Evaluation of Xtreme Reading Implementation Sean Murphy North East Leadership Academy Cohort 1 Spring 2012, NCSU

2. Problem of Practice STRATEGIES ULTIMATE GOAL OBJECTIVES A: Train teachers on the 13 instructional stages used year long as core instructional strategies Teachers know and use research-based strategies effectively Teachers know how to reach struggling learners Teachers differentiate instruction B: Train teachers on research based literacy strategies through Xtreme Reading Program At-risk students are more confident and successful readers. Students collaborate during class C: Train teachers to use Possible Selves strategy Students use literacy strategies proficiently D: Provide on-going support forXtreme reading teachers and students

3. STRATEGIES Evaluation Questions Data How many hours were spent training teachers on 13 stages of instruction? • Training agendas A: The 13 instructional stages provided an instructional routine of pedagogic practice (the “how”). How many hours were spent supporting teachers on the 13 stages of instruction? • Support logs • Professional Development Questionnaire Did the teachers think the session was of high quality? On how many strategies were teachers trained? • Training agendas B:Xtreme Reading included 8 research-based literacy strategies explicitly taught to students (the “what”). How many professional development training sessions were given? • Support logs • Professional Development Questionnaire To what extent did teachers think the trainings were useful for their instruction? C: Possible Selves was a motivational strategy. For the purpose of the project, results were insignificant . D: On-going support for Xtreme Reading teachers and students included coaching, planning assistance and modeling. How many hours of support were provided for the strategies? Likert Scale How supported did teachers feel during the process? Focus Group

4. Data Objectives Evaluation Questions How confident do teachers feel using research-based strategies? 1: Teachers can explicitly teach literacy strategies to students (the “what”). How often do teachers use research-based strategies? • Professional Development Questionnaire • Teacher Self-Report Survey 2: Teachers differentiate instruction, as laid out in the 13 Instructional Stages (the “how”). • Learning Strategies Walk Through Rubric How often do teachers differentiate instruction? • Retrospective Pre/Post Student Survey • Pre-Post Assessments for Each Strategy 3: Students collaborate during class, as laid out in the 13 instructional stages (the “how”). How often do students collaborate during class? 4: Students use literacy strategies proficiently How many students used the strategies proficiently?

5. Results A: The 13 instructional stages provided an instructional routine of pedagogically practice (the “how”) • According to training agendas, teachers received approximately 4 hours of training on the 13 stages of instruction at the beginning of the year. • According to the support logs, teachers received 7 hours of support specifically regarding teacher practices (13 stages of instruction). Teachers did not receive support for 13 stages of instruction (as a whole model) after initial training. However, there was specific support for differentiation and student collaboration. For the former teaching strategy, I provided 3 hours of support (mainly through directive support and modeling) and curriculum coaches provided .5 hours. Teachers received 1 hour of support on student collaboration from myself and 2.5 hours of support from coaches.

6. Results B: Xtreme Reading included 8 research-based literacy strategies explicitly taught to students (the “what”) • Teachers were trained on a total of 3 strategies during the first semester of implementation, conducted over 4 training sessions. Teachers received a total of 12 hours of training each during the Fall Semester. Teachers were trained on 2 strategies from January through March in 3 training sessions, totaling 8 hours for each teacher. • Over the course of the semester teachers were trained on literacy strategies. Their reaction was mixed. During the first session, 1 out of 3 teachers strongly agreed that training was useful for their instruction, 2/3 disagreed. For the second session, 1/3 strongly agreed and 2/3 agreed. With the third session, all teachers strongly disagreed that it was useful. All agreed the fourth session was useful.

7. Results

8. Results C: Possible Selves was a motivational strategy. • Questionnaires showed that 3/3 teachers felt confident and prepared to teach the Possible Selves strategy after training. • According to interviews, teachers were introduced and discussed how to use Possible Selves strategies in the classroom. Sharing of Possible Selves lessons was facilitated by administration during weekly PLC sessions. Only 1 lesson was shared, student artifacts were collected but not reviewed.

9. Results D: On-going support for Xtreme reading teachers and students included coaching, planning assistance and modeling. • Support logs showed administration provided 26 hours of support either individually or with Xtreme teachers as a PLC, over the course of the school year. Support came in the form of coaching, modeling and facilitating planning. University of Kansas coaches provided 6 hours. Support included coaching and facilitating planning. • According to self-report likert scales, 3 out of 3 teachers strongly agreed that administrators prepared them for implementing strategies. They also strongly agreed that administration supplied adequate on-going support to effectively teach strategies. However, the feelings about curriculum coaches were less enthusiastic. 2 teachers agreed that the coaches prepared them for implementing strategies and 1 disagreed. Regarding the on-going support from curriculum coaches, 1 teacher felt neutral about the level of support and 2 felt inadequately supported. • In a 30-minute focus group, teachers voiced negative feelings toward support 18 times and positive feelings toward support 6 times. It was stated that more on-going support and coaching was needed after initial training. Specifically, teachers felt “neglected,” “abandoned,” and “used” by the curriculum coaches after training. They generally agreed that administration provided on-going support (4 of 6 positive comments were directed toward administration).

10. Results 1: Teachers can explicitly teach literacy strategies to students (the “what”) • Based on survey data, 66% of teachers they strongly agreed that they were confident teaching three out of four of the strategies they received training. They were least confident teaching the Word ID strategy. In comments, teachers said this training was ineffective because it was conducted virtually through Face Time link. • 26 copies of the Learning Strategies Walk Through rubric showed that teachers were using research-based strategies on 12 occasions; 4 in January, 6 in February, 2 in March.

11. Results 2: Teachers differentiate instruction, as laid out in the 13 Instructional Stages (the “how”) • According to 26 copies of the Learning Strategies Walk Through Rubric, teachers differentiated instruction 18 times; 2 in January, 6 in February and 10 in March. Differentiation consisted of purposeful organization of group work, leveled texts, task variety and individual support.

12. Results 3: Students collaborate during class, as laid out in the 13 instructional stages (the “how”) • According to a retrospective pre/post survey, Xtreme Reading students said the number of times they collaborated in the classroom increased from the first days of school to the middle of spring. Specifically, in response to a prompt asking, “how often did you collaborate the beginning of the school year,” 33% responded “about once a week”, 29% responded “once or twice a week” and 14% collaborated “everyday.” In the same survey, a prompt asked student, “how often do you collaborate now?” 20% of students marked that they collaborated “once a week,” 42% reported “once or twice a week” and 25% said they collaborated everyday. • In 26 copies of the Learning Strategies Walk Through Rubric taken from December-March, students were collaborating 18 times. Collaboration consisted of student paired-reading 8 times and collaboration on projects 10 times. 2 in January, 6 in February, 10 in March.

13. Results

14. Results

15. Recommendations: Strategies • Provide More Spiraled Support for 13 Stages of Instruction: One major lesson learned, in reflection, was the necessity for clarifying the need for instructional change. One of the reasons Xtreme Reading was chosen as an intervention program was that it required differentiated and collaborative work. At the time of initial training it was assumed that this was already a routine, to some extent, in the classrooms. Thus, teachers did not have ample training on effective pedagogy, as presented in the 13 instructional stages routine. Informal and formal observations showed that South Johnston teachers relied heavily on direct instruction and assessment. This was true both before and after the program, proving the need to measure the amount of differentiated instruction and collaborative work. In preparing for next year’s implementation, much consideration and focused support should be given on instructional change. • Offer more support. Three major changes should be made in supporting methods: 1) Frontload support, 2) Offer more ongoing support, 3) Make support more directive. First, after consulting with the KU coaches, it is clear that amount of support should be more frontloaded next year, with consistent visits and updates during the first quarter. Second, while the amount of support might decrease in the second semester, open and regular dialogue should persist throughout the implementation (including analysis of data collected). The third point will be discussed in more detail below. At this point, though, it is worth mentioning that support should be more rooted in the materials and include specific follow-up about how instruction can be improved. • Manage the Transition. In hindsight, it is clear that the transition from should have been planned and managed more effectively. Many of the teachers made it clear in support sessions that they were unfamiliar with specific strategies. They voiced apprehension with teaching reading strategies explicitly and had a hard time letting go of traditional instructional practices (predominately lecture and test). This could be expected, but plans should have including pathways forward, including methods for celebrating accomplishments and acknowledging areas for improvement.

16. Recommendations: Objectives • Utilize the program’s resources. This was alluded to above in the “offer more directive support” bullet. The Xtreme Reading program has a vast amount of resources, including lesson plans, pacing outlines, and formative assessments that can inform instruction. Yet, almost all of the support we offered (especially in the beginning of the year) did not put these materials to use. On a regular basis, administration and/or instructional coaches should sit down with the teachers and the materials. This can be used to ground conversation. Teachers should reflect on how closely their planning and pedagogy follow the guidelines. Of course, teachers should have the autonomy to make modifications; however, they must know what the standard looks like and try adhering to the program as it is designed. • Make Goals Clearer and More Prominent: In retrospect, the outcomes for the program should’ve been framed in both teacher and student terms. We were trying to change instructional habits as much as content. However, this is a more delicate process, as former teaching habits were already in place, demanding attention to the transition. What’s more, the entire instructional cycle should have been emphasized, so that teachers were constantly reminded how differentiated teaching and collaborative learning fit into the framework for teaching. Once the goals were stated more clearly, systems should be in place to regularly monitor progress. The walk through form could have been used more consistently with teachers, anchoring coaching conversations.

17. Evaluation Reflections • Don’t ask so many questions. To be embarrassingly honest, I didn’t mean for this PowerPoint to be this long. However, by asking so many questions, I left myself with little choice. I remember Dr. Corn saying, “Keep it simple, you’ll have to collect data on everything you ask.” Well, I guess we learn from experience. • Stop and smell the data. I waited too long to really start analyzing the data I was collecting. If I had started earlier, I would’ve noticed a pattern (content not changing, pedagogy changing) earlier on. This could have shaped student interventions and teacher support. • Sense of efficacy from understanding the process. In general, I didn’t have the slightest idea about how to conduct an program evaluation. This projects helped me take the long-view. By seeing how the entire cycle is completed, I’ll be more prepared every step of the way, from planning to reflection.