West Mifflin Area School District An Analysis of 4 Sight and How It Is Used to Inform Instruction

1. West Mifflin Area School DistrictAn Analysis of 4 Sight and How It Is Used to Inform Instruction Dr. Bart Rocco, Assistant Superintendent Dr. Daniel Castagna, Principal, Homeville Elementary

2. Today’s Presentation Role of Central Office in Using Data to Make Informed Decisions to Improve Student Performance Four Key Points to Process A Quantitative Case Study Analysis of the 4 Sight Benchmark Assessment Implications Questions/Answers

3. West Mifflin Area School District 3,400 Students 4 Elementary Schools,early Childhood Center, Middle School, High School Pre-K- 12 Program 45% of Students Economically Disadvantaged 40% Special Education Population AYP Status of Schools

4. How did we get here? Two years ago no formalized benchmark assessments Reviewed Benchmark Assessments Approved for Grants Developed a Five-Year Plan Classroom Excellence Initiative Ed Insight Technology Limitations/Needs Curriculum Alignment Process

5. Never Enough Time Rearranged professional development time Act 80 2 hour delays Insisted on before school/after school grade level, and department level meetings to address student data as a continuous process 10,000 hour rule – Outliers, Malcolm Gladwell More time with students

6. Question Data/Hypothesize Review data and develop focused questions Look at specific student performance on tests in math and reading Examine attendance Look at target populations What are root causes? Speak to previous year’s teachers about student performance

7. Best Practice Staff’s need to examine best teaching practices What is the best way to teach a concept in developing main idea or a specific math skill Develop a toolbox of activities Share Ideas with each other. Replicate testing experience

8. Leadership Principals need to allocate time for collaboration Put data into the hands of teachers Communicate to your staffs and support them in their needs Be engaged in instruction and learning

9. Leadership Curriculum Alignment to Standards Create a balance between improving test scores and maintaining a culturally enriched curriculum Don’t forget about what standardized tests cannot do for our students Build trust with your administrators Work the vineyard

10. Thanks to the Classroom Excellence Initiative • Debbie Raubenstrauch • Jim Turner • Steve Biancaniello • On Hands School

11. Why use the 4Sight tests???

12. Growing Popularity !!! Source: taken from Pennsylvania and 4Sight, newsletter published by Success for All, October 2007.

13. Arguments arose… • Are the predictions accurate? • Is the money worth it? (over $10,000 annually) • Are we giving in to over-testing? • Why are PVAAS projections lower than 4Sights? • Are the skills tested on the 4Sight correlated to those tested on the PSSA

14. THE STUDY….

15. Primary Research Questions… • Do the 4Sight exams accurately predict student’s raw scores on the PSSA exam? • Is there a high correlation between the predicted categorical classifications of students on the 4Sight tests when compared to actual PSSA Results? • Is the information gained from the fourth 4Sight test worth the loss of instructional time?

16. Data Collection Procedures Four Benchmark Tests were Administered: • 4Sight Test #1 – September 11, 2007 • 4Sight Test #2 – November 22, 2007 • 4Sight Test #3 – January 12, 2008 • 4Sight Test #4 – April 14, 2008 • PSSA Exams – April 1, 2, 3, 2008 • West Mifflin Middle School Students in Grades 6, 7, and 8. • 686 out of 731 qualified for the study (94% of total population).

17. Data Table Example…

18. Data Analysis Procedures • Linear Regression with an (r) value. • The independent variable- 4Sight Score (X) was used to predict the dependent variable- PSSA score (Y) • In the January/February 2008 Newsletter published by the Success for All Foundation it was stated that, “The 4Sight correlations are determined by a statistical process called linear regression” (p 2). • Kappa Coefficient Formula. • Measured the agreement between predicted categorical classifications of students and actual PSSA categories.

19. Interpretation of Results… Linear Regression • The (r) score can range from 0 to 1 with scores closer to 1 showing a stronger correlation and scores close to 0 showing a weaker correlation. Kappa Coefficient • Landis & Koch (1997) suggests the following kappa interpretation scale: • Below 0.00 - Poor • 0.00-0.20 - Slight • 0.21-0.40 - Fair • 0.41-0.60 - Moderate • 0.61-0.80 - Substantial • 0.81-1.00 - Almost perfect

20. Regression Results - 4Sight Score (X) was used to predict the dependent variable- PSSA score (Y)

21. Kappa Coefficient Results

22. Kappa Coefficient Results

23. Findings: • Do the 4Sight exams accurately predict student’s raw scores on the PSSA exam? • Is there a high correlation between the predicted categorical classifications of students on the 4Sight tests when compared to actual PSSA Results? • Is the information gained from the fourth 4Sight test worth the loss of instructional time?

24. The Model…

25. What were the results… MATH

26. What did we do with the results? • All teachers were given a copy of updated anchors for Reading and Math. • Open ended items scored by teachers of the school. • Department meetings were held to review data and identify trends across students. • Item analysis breakdowns for each question (and corresponding anchor) were provided for all teachers.

27. 5. All data was uploaded into EdInsight software so teachers could view strengths and weaknesses of students in all of their classes. Appendix C (also included-breakdown by reporting category)

28. Flexible Groups for MathChart 6. Flexible groups were created within all subjects to highlight students with common areas of concern. Teacher________________________ Subject________________________ Period_________________________

29. Flexible Groups Allow us to… • Assess strengths and weaknesses in light of the PA Reporting Categories and Academic Anchors • Develop strategies to address concernsat both class and individual student levels • Use this information to plan large group (across grade level). • TEST NARROWLY BUT TEACH BROADLEY

30. Most Importantly… With this degree of information, teachers can more easily embed these skills and knowledge in the larger curriculum.  SO…. Classroom curriculum is enhanced rather than narrowed! We are not teaching to the test – We are teaching to the needs of our students!

31. Unintended benefits! • Practice run for the PSSA testing. Beneficial for both students and staff! • Students get use to the “testing room”. • Scheduling issues were worked out (what do we do with teachers who do not have homerooms?). • We have enough pencils, paper and calculators!!! • Students report that they were less nervous for the 2nd round of the 4 Sight tests. • Found/identified building level trends in open-ended responses. Has opened communication between departments. • Created incentives for attendance and participation (grade level challenges!).

32. Better use of Support Resources • EdInsight Software • Skills Tutor Software • Study Island • PSSA Coaching Books

33. What Happened???2008 PSSA Results…

34. Ideas for Future Research • Investigate a longitudinal study that follows the use of the 4Sight test with a cohort of students over several years. • Identify a model of best practices to use when implementing 4Sight assessments. • Compare specific classroom interventions used in conjunction with benchmark assessment with student achievement growth.

35. Ideas for Future Research • Expand this study to include all students in grades 3-8 and 11. • Narrow the focus of the study to a particular sub-group or groups. • Analyze specific skills assessed on the PSSA and compare them with skills tested on the 4Sight exam.

36. Questions?