Examining Perceptions of Success During the Implementation of an Instructional Policy

1. Examining Perceptions of Success During the Implementation of an Instructional Policy Dr. Kathryn Ohle University of Alaska Anchorage Literacy Research Association Wednesday, December 4th, 2013

2. Today I will: • Describe my approach • Explain my methods • Share my findings • Elaborate on points of interest • Discuss potential limitations • Outline where I would like to go next • Answer any questions

3. The Context – The Targeted Reading Intervention (TRI) • First grant received in 2005 - both a goal 2 (development) & goal 3 (efficacy) via IES Rural School and Community Trust Grant #R305A10000654-11; latest grant (2010) focuses on goal 3 via IES Grant#R305A040056. • A randomized control trial that uses a dual-level professional development intervention for at-risk K-1 students and classroom teachers. • Teachers learn skills that increase knowledge of early reading skills, improve instructional strategies, and match instruction to informal assessment. They also receive PD in early reading practices during periodic workshops and weekly web-coaching sessions with a coach • Children receive individualized daily instruction in 15-minute lessons that address fluency building, decoding skills, and guided oral reading. • Child measures include DIBELS, Woodcock Johnson (5 subtests), & CTOPP (5 subtests). Under the direction of Dr. Lynne Vernon Feagans (PI) and Dr. Mary Bratsch-Hines (Program Director)

4. TRI – 3 Studies’ RCT Outcomes

5. The Research Questions for this Study: How do stakeholders measure the “success” of the TRI? Do stakeholders’ perceptions of a policy’s success differ within or between groups? How so? How do these perceptions of success impact the implementation of the policy?

6. My Theoretical Approach Examine TRI as a policy “An unofficial, nongovernmental, or informal practice that shapes behaviors and outcomes” (Marshall & Gerstl-Pepin, 2005. p.37)

7. My Theoretical Approach Pay attention to the perceptions of key implementers of the policy, within context “Implementers at all levels of the system effectively negotiate their response, fitting their action to the multiple demands, priorities, and values operating in their environment and the effective authority of the policy itself” (McLaughlin, 1987b, p.175).

8. Methods • Qualitative methods – selective and open-coding, constant comparison method • Primary source of data – semi-structured interviews • 10 teachers, 2 principals, 5 researchers, 1 literacy coach • Secondary source of data – focus groups, field notes, e-mail correspondences

9. Approach to Analysis Step 1 – Apply intentional codes to data gathered through interviews Step 2 – Organize answers into data displays Step 3 - Look for evidence of codes identified in steps 1 and 2 in secondary data sources; add to data displays Step 4 – Allow for open-coding to allow for additional patterns or themes to emerge Step 5 – Identify commonalities and differences within and between groups of stakeholders and explore reasons for variations (informed by literature & critical theory) Step 6 – Summarize findings Step 7– Apply both traditional and critical policy analysis

10. How do stakeholders measure the “success” of the TRI? Teachers said, “Student Growth” • Standardized test scores “I just wanted to share some great news - I tested JA today on running records and she passed a level 14!!! She was an 8 in January before we started. I was so excited - along with her mother!!” (Nellie, e-mail communication, March 16, 2012). “I am sooooooo excited to write and tell you that B (student) went from an “E” to an “I” this nine weeks!!!!!!!!!” (Tonya, e-mail communication, May 21, 2012).

11. How do stakeholders measure the “success” of the TRI? Teachers said, “Student Growth” • Anecdotal notes related to students’ academic growth “He really is moving right along and I've been impressed with his word skills. I still see some weaknesses in fluency and sight words, but I see lots of improvements” (Lauren, e-mail communication, December 21, 2011). “I thought Josie Anne was going to be retained and never could have gotten her to grade level without TRI!” (Nellie, interview response, August 17, 2012)

12. How do stakeholders measure the “success” of the TRI? Teachers said, “Student Growth” • Student growth in confidence and motivation “My kids went from rolling their eyes and saying ‘I can’t read’ to actually reading and beating me to the reading table to do it” (Lauren, interview response, August 15, 2012). “When the child who hated reading was now asking if he could read to others, I knew it [the TRI] was a success” (Maddie, interview response, August 9, 2012).

13. How do stakeholders measure the “success” of the TRI? Teachers said, “Teacher growth” • Content knowledge “On a personal level, it was a wonderful resource for me, to guide me through my first year of first grade. Helped me know how to help my children” (Gloria, interview response, August 13, 2012). • Confidence “Hey, I can really do this” (Pam, interview response, August 13, 2012). “TRI has changed my perceptions on how long it takes a struggling child to catch up, as I’d always heard it took two years but now I see that it takes shorter” (Maddie, focus group response, August 9, 2012)

14. How do stakeholders measure the “success” of the TRI? Researchers said, “Student Growth” • Standardized test scores “if they had made big gains on standardized reading tests, they had hit the gold standard” (Lila, interview response, August 15, 2012) “I have heard anecdotally from the coaches in team meetings the teachers were successful but some anecdotal evidence shared by the assessors [who administered the battery of tests like the Woodcock-Johnson] indicated some teachers were disheartened. We will just need to see data, mainly the pre and post test scores of the kids as represented through statistics” (Alicia, interview response, August 13, 2012).

15. How do stakeholders measure the “success” of the TRI? Principals said, “Student Growth” • Standardized test scores (Dibels & running records) Literacy coach said, “Teacher efficacy” & “Student Growth” • Confidence levels, often they did it, instructional match • Running record levels, end-of-the-year outcomes

16. Do stakeholders’ perceptions of a policy’s success differ within or between groups? How so? • Teachers – focus on student achievement scores (measured through running records), increased levels of student motivation and confidence, and teacher growth and self awareness • Researchers – focus on student achievement scores (DIBELS & WJ scores), some focus on fidelity • Principals – focus on student achievement scores (reading levels and DIBELS) • Coach – focus on teacher capacity and long-term outcomes

17. How do these perceptions of success impact the implementation of the policy? • Affects motivation & buy-in “If students, teachers, or administrators believe that the results of an examination are important, the effect is produced by what individuals perceive to be the case” (Madaus, 1999, p.78 as cited in McGee, 2006). The expectation is that once the teachers see change, they will feel successful and motivated to continue implementing the TRI on a regular basis, as activities that are successful are more likely to be repeated than those that are not (1986). If the teachers do not see a change, especially in student outcomes (i.e. don’t feel successful), there is less inducement for them to change their beliefs, attitudes, or teaching practices (Lamb, Cooper, & Warren, 2007).

18. Points of Interest • Factors that may have impacted the results: • Perceptions of roles • Perceptions of real and perceived lack of control • Perceptions of influence • Perceptions of responsibility • Power dynamics amongst stakeholders

19. Limitations • Interviewing my own teachers • Focus on perceptions means subjective answers • Small sample size limits any type of generalizability, especially with principals • Influences on continued implementation and teacher buy-in (interviews only after year 1)

20. Future Directions • Add in a Mixed Methods design and compare standardized assessment results with teacher perceptions • Broaden the group of stakeholders to include parents and students • Revisit after Year Two of implementation • Add in coach perceptions in a more deliberate manner – my influence was obvious!

21. Questions? Please feel free to contact me at: kohle@uaa.alaska.edu