1 / 24

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

School Improvement Network Impact Assessment: Higher Engagement Schools versus Lower Engagement Schools. Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator December 2012. Overarching Research Question:. Does engagement

gail
Download Presentation

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. School Improvement Network Impact Assessment:Higher Engagement Schools versus Lower Engagement Schools Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator December 2012

  2. Overarching Research Question: Does engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success and school-wide metrics?

  3. Sample Description • High Video Utilizers • 39States • 211 Districts • 734 Schools • Metrics: • Educator Engagement • Student Success • School Impacts

  4. Study of Educator Engagement • 32 data elements collected or computed through PD 360 and Observation 360 • Contrasted higher engagement schools versus lower engagement schools • Improvement in percentages of students who tested advanced or proficient in math and reading • Classified into four quartiles • Analyses of highest and lowest quartiles only

  5. Metrics for Differentiating Advantages for Higher Engagement Organizations: Leadership, Implementation and Accountability • Focus Objectives Set Up • Observations Performed • Percent Registered Users • Percent of Users in Communities • Minutes Viewed • Forums Viewed • Programs Viewed • Segments Viewed • Links Viewed • Follow-up Questions Answered • Reflection Questions Answered • Focus Objectives Set Up • Forums Posted • Downloaded Files • Uploaded Files • Participation in Communities Educator Participation Educator Engagement These are the 15 metrics for which higher engagement schools were significantly higher than their lower engagement counterparts

  6. Sample of Differentiating Metrics of Utilization and Engagement Passive participation (e.g. video viewing alone) is LESS influential than Active engagement Links Viewed Observations Performed 70.3% advantage (p<.001) 63.8% advantage (p<.001) Minutes Viewed Follow-up Questions Answered 4.3% advantage (p<.001) 39.0% advantage (p<.001)

  7. Sample of Differentiating Metrics of Utilization versus Engagement Passive participation (e.g. video viewing alone) is LESS influential than Active engagement Downloaded Files Forums Viewed 30.5% advantage (p<.001) 79.5% advantage (p<.001) Uploaded Files Forums Posted 68.6% advantage (p<.001) 47.3% advantage (p<.001)

  8. Who Cares? Who cares if educators used it more? Did it make a difference for kids and schools?

  9. Study of Student Success Student Success: • Performance on standardized tests • Percent either proficient or advanced in the following subjects: • Reading • Math

  10. ImprovedStudent Performance 4.9% gain for lower engagement schools (p<.01)

  11. ImprovedStudent Performance Closed the Gap: 267% advantage in gains for higher engagement schools (p<.001) Nearly 4 times the impact 4.9% gain for lower engagement schools (p<.01) 18.0% gain for higher engagement schools (p<.001)

  12. ImprovedStudent Performance 0.5% gain for lower engagement schools (p=ns) Actually Important Gains: For every 200 students, 1 more performed at proficient or advanced level than in the previous year

  13. ImprovedStudent Performance Surpassed the Gap: 3,520% advantage in gains for higher engagement schools (p<.001) 36 times greater impact 0.5% gain for lower engagement schools (p=ns) 18.9% gain for higher engagement schools (p<.001)

  14. Metrics of School Impact • Performance on key indicators from Internet (when publicly available) and structured telephone interviews: • Dropout Rates • Student Discipline Rates • Teacher Retention Rates • College-Bound Rates

  15. Improved Dropout Rates 4.9% improvement for lower engagement schools (p<.01) For every 100 students, 5 fewer dropped out than in the previous year Figures reflect rounding, projections reflect correct math.

  16. Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) Improved Dropout Rates 309.1% advantage in gains for higher engagement schools (p<.001) 4.9% improvement for lower engagement schools (p<.01) 20.0% improvement for higher engagement schools (p<.001) For every 100 students, 20 fewer dropped out than in the previous year.

  17. Improved Student Discipline Rates 7.4% fewer disciplinary incidents for lower engagement schools (p<.01) Figures reflect rounding, projections reflect correct math. Y-Axis is inverted to reflect improvement as intuitively upward trend.

  18. Improved Student Discipline Rates Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) 351% advantage in gains for higher engagement schools (p<.001) 4 ½ times the impact 7.4% fewer disciplinary incidents for lower engagement schools (p<.01) 33.2% fewer disciplinary incidents for higher engagement schools (p<.001) For every 100 students, 33 fewer problem students than in the previous year.

  19. ImprovedTeacher Retention Rates 1.7% more teachers stayed for lower engagement schools (p<.01) Figures reflect rounding, projections reflect correct math.

  20. ImprovedTeacher Retention Rates Executive Summary Higher engagement schools significantly outperformed their counterparts (p<.01) 65.9% advantage in gains for higher engagement schools (p<.001) Nearly twice the impact 1.7% more teachers stayed for lower engagement schools (p<.01) 2.8% more teachers stayed for higher engagement schools (p<.01) For every 100 teachers, nearly 3 fewer left than in the previous year. Figures reflect rounding, projections reflect correct math.

  21. Improved College-Bound Rates Percentage of students schools report as being college-bound. No decrease or gain for lower engagement schools (ns) Figures reflect rounding, projections reflect correct math.

  22. Executive Summary Higher engagement schools began statistically equal, then significantly outperformed their counterparts (p<.01) Improved College-Bound Rates Incalculable advantage in gains for higher engagement schools (p<.001) No decrease or gain for lower engagement schools (ns) 9.6% improvement for higher engagement schools (p<.001) For every 100 students, 10 more were college-bound than in the previous year. Figures reflect rounding, projections reflect correct math.

  23. Summary of School Impacts • Dropout Rates • Approx. 15 fewer dropouts per 100 students than lower engagement school counterparts • Student Discipline Rates • Approx. 33 fewer students “in the office” per 100 students than for lower engagement school counterparts • Teacher Retention Rates • Approx. 3 fewer teachers leaving per 100 teachers, which is 1 fewer than for lower engagement school counterparts • College-Bound Rates • Approx. 10 more college-bound students per 100 students than for lower engagement school counterparts

  24. A Model for Educational Success Leadership, Implementation and Accountability Student Success Educator Participation School Impacts Educator Engagement

More Related