An Examination of Middle School Problem-Solving Teams

1. An Examination of Middle School Problem-Solving Teams Nicki Meshbesher, Ph.D. Dissertation May 2010

2. Why Study Problem Solving-Teams? • Proliferation of PSTs within schools (Buck, Polloway, Smith-Thomas, & Cook, 2003; Truscott, Cohen, Sams, Sanborn, & Frank, 2005; Carter & Sugai, 1989) • PSTs characterized by inconsistent implementation(Buck et al., 2003; Burns, Vanderwood, & Ruby, 2005) • Important to evaluate whether PSTs are helping schools achieve desired outcomes

3. Why study PSTs at Secondary Level? • Paucity of research at secondary level (Burns & Symington, 2002; Telzrow et al., 2000; McNamara et al., 2008) • Qualitative differences between elementary & secondary schools

4. Current Investigation • Unique because focused solely on middle schools • Used multiple sources of data to address limitations from previous studies

5. Purpose of Investigation • Examined a model of middle school grade level PSTs • Examined fidelity of implementation of Problem Solving Process (PSP) used by middle school KT teams • Examined goal attainment for students referred to and receiving assistance through KT teams • Examined KT team member perceptions of student outcomes and satisfaction with their KT team

6. Research Questions RQ 1 With what degree of fidelity did the middle school Kid Talk (KT) teams implement identified key components of the problem solving process (PSP)? RQ 2 To what degree did students who were referred to the middle Kid Talk (KT) teams attain targeted academic or behavioral goals? RQ 3 What was the relationship between fidelity of implementation between each of the key components of the problem solving process (PSP) and student goal attainment? RQ 4 What were Kid Talk (KT) team member ratings of satisfaction of the KT team process and perceptions of student outcomes for students referred to and receiving interventions from the KT team?

7. Participants • 16 middle schools within local district • Case documentation (59 cases) from 16 schools • Process observations of 3 selected KT teams • Survey respondents (286 KT team members across 16 schools)

8. Instruments • PSP components scoring rubric (modified from Telzrow et al. 2000) • 8 PSP components measured on 5-point Likert scale (1 = low fidelity; 5 = high fidelity) • Goal attainment measured on 5-point Likert scale (1 = progress away from baseline; 3 = progress made toward goal but not met; 5 = interim / long-term goal met or exceeded) • Student Outcomes and KT Team Survey • Online electronic 18-item survey • Multiple choice items • 5-point Likert scale (1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree) • Process Observation Form (POF) • 13 items scored as “yes” or “no”

9. Limitations • Subjects • Convenience sample affects generalizeability • Availability of data • “Real world” implementation led to challenges accessing data • Case documentation • Unable to get amount of cases requested or random cases • Variation of case documentation • Using case documentation as measure of fidelity • Instrumentation • Response bias

10. ResultsDescriptive Data Case Documentation • 59 cases across 16 middle schools • Cases per school M= 3.69; range per school = 2-5 • Cases by grade level (Gr 6 = 25; Gr 7 = 20, Gr 8 = 14) • Types of concerns across all cases • General academic = 40.2% • Specific academic = 20.7% • Behavior = 31.8% • Attendance = 7.3%

11. ResultsDescriptive Data Student Outcomes and KT Team Survey • 286 respondents from 16 middle schools • Response rate across schools (ranges 6.9% - 97.2%) • Demographic information • General Educators, (ranges 66.7%-87.5% across schools) • Gender, mostly female (ranges 60.0% - 85.0% across schools)

12. ResultsSummary • PSP components not implemented as prescribed • Mean ratings range from 1.78 (intervention skill development) to 3.48 (baseline data) • 1 = low fidelity; 5 = high fidelity • Low number of cases that yielded student outcomes • N=23 (38.9% of cases could be scored) • 1 = trend away from baseline; 3 = progress toward goal; 5 = Interim or Long-Term goal met or exceeded • Neutral to modest support from KT team members • KT team members report positive comfort level referring students to their team, indicative of “comfortable collaboration” • Neutral support indicating that the student improved after receiving interventions from KT Team or that goals of the intervention were accomplished

13. Discussion Why the less than desired levels of fidelity of implementation? • Different PSP models specified by district • Variability in practices within and across schools • Lack of clarity leads to different innovation configurations (Hall & Hord, 2006) • 16 participating schools (7 used SDF, 6 used Action Plan, 3 used both; Action Plan forms not consistent within or across schools) Is the KT team process resulting in desired student outcomes? • Low production cases (<39% cases could be scored) • Superficial concerns identified, no drilling down (>70% cases were general academic or behavior) • EOY data 110 cases across all schools, 71.8% came from 5 schools (79 cases). Remaining schools had on average <3 cases with student outcome data for entire year

14. Discussion • What do KT team members think? Comments from survey respondents • “Too much narration and storytelling takes place, and too little actual consideration of the problem and interventions takes place.” • “Teachers are notorious storytellers. We all want to tell the funny / irritating anecdote. However, KT needs to be more focused on how to fix things.” • “An extremely slow process and often we go in circles.” • “[The] kid talk process tends to be time consuming based on the results. A flawed system. I won't call it a complete waste of time but for all the effort invested, the outcomes don't even come close to providing any satisfaction to the staff or the student involved.”

15. Implications for practice • Reconsider PSP model and clearly specify PSP model, establish ownership, and provide support • System and admin support critical (Fuchs et al., 1996; Santangelo, 2009) to achieve and sustain desired levels of fidelity • Shared vision, purpose and understanding by all stakeholders critical • Training of specific skills is critical but establishing a clear purpose and ownership of PSP model primary

16. Implications for Practice • Questionable efficacy of current team format due to challenges of team-based problem solving • Group / team dynamics and functioning affected by social influences (e.g. verifying personal experiences through storytelling, team members opinions converge to create a “shared reality”, Higgens, 1999) • Communities of practice theory as a framework, Benn (in press) found negative impact of social influences resulting in team members failing to make needed conceptual shifts • Hard to truly be reflective about instructional practices • Teacher requests for assistance often perceived as lacking professional competence

17. Implications for Practice • Differences at middle school level make PST implementation more challenging, need to consider supports needed • More student independence required • Demands to meet established standards • Teachers more focused on content • Teachers responsible for more students

18. References • Benn, A. (2004). Communities of Practice: Study of one school's first year of implementation of a new problem-solving model. (Doctoral dissertation). ProQuest. (AAT 3178485). • Buck, G. H., Polloway, E. A., Smith-Thomas, A., & Wilcox-Cook, K. (2003). Prereferral intervention processes: A survey of state practices. Exceptional Children, 69, 349-360. • Burns, M. K., & Symington, T. (2002). A meta-analysis of prereferral intervention teams: Student and systemic outcomes. Journal of School Psychology, 40, 437-447. • Burns, M. K., Vanderwood, M. L., & Ruby, S. (2005). Evaluating the readiness of pre-referral intervention teams for use in a problem solving model. School Psychology Quarterly, 20, 89-105. • Carter, J., & Sugai, G. (1989). Survey on prereferral practices: Responses from state departments of education. Exceptional Children, 55, 298-302. • Fuchs, D., Fuchs, L. S., Harris, A. H., & Roberts, P. H. (1996). Bridging the research-to-practice gap with mainstream assistance teams: A cautionary tale. School Psychology Quarterly, 11, 244-266.

19. References • Hall, G. E., & Hord, S. M. (2006). Implementing change: Patterns, principles, and potholes, 2nd ed. Boston: Pearson. • McNamara, K., Rasheed, H., & Delamatre, J. (2008). A statewide study of school-based intervention teams: Characteristics, member perceptions, and outcomes. Journal of Educational and Psychological Consultation, 18, 5-30. • Santangelo, T. (2009). Collaborative problem solving effectively implemented but not sustained: A case for aligning the sun, the moon, and the stars. Exceptional Children, 75, 185-209. • Telzrow, C. F., McNamara, K., & Hollinger, C. L. (2000). Fidelity of problem- solving implementation and relationship to student performance. School Psychology Review, 29, 443-461. • Truscott, S. D., Cohen, C. E., Sams, D. P., Sanborn, K. J., & Frank, A. J. (2005). The current state of prereferral intervention teams. Remedial and Special Education, 26, 130-140.