1 / 26

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

Program Evaluation and Impact Assessment: Internet-Delivered, On-Demand Professional Development Participating Schools versus Their Respective Districts. Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator July 2013.

hanna-riggs
Download Presentation

Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Program Evaluation and Impact Assessment: Internet-Delivered, On-Demand Professional DevelopmentParticipating Schools versus Their Respective Districts Steven H. Shaha, PhD, DBA Professor, Center for Public Policy and Administration Independent Evaluator July 2013 NOTE: Findings that follow are published as: Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013 (in press)

  2. Overarching Research Question: Does teacher engagement in PD 360 and Observation 360, tools within the Educator Effectiveness System, significantly affect student success?

  3. Methods • Design: Quasi-experimental, retrospective, pre-post, normalized treatment-control / participation vs. non-participation (2009-10, 2010-11) • Goal: Multi-State, large n with comparable student populations (matched, controlled) • Student Change:*Metric was percent students classified as Proficient or Advanced in respective States. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  4. Sample Participation • Systematic sample of 169 elementary schools, in 73 districts, in 19 States • N determined by a priori Power analysis • Schools eligible for inclusion in the sample as participating Schools met the following criteria: • More than 10 teachers total • 80% or more of teachers viewed materials • Minimum average of 90.0 minutes of viewing per teacher for the school • Districts included were only those for which eligible schools were included • Normalizing for difference in socio-economic and demographic factors between participating Schools and their Districts cumulatively as the statistical comparison group Data • Participation data were extracted from the Internet-based professional development application as surveilled • Student performance data were captured from publically available, Internet-accessed sources (school as unit of measure, percent Proficient or Advanced as metric) Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  5. Impacts on Math Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  6. Impacts on Math Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  7. Impacts on Math Districts improved by 2.6 net percentage points (p<.01). Improvement is percent change: [(Year2-Year1)/Year1] 7 Comparative change: [School change/District change] Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  8. Impacts on Math Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  9. Impacts on Math Participating Schools improved by 11.1 net percentage points (p<.001). Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  10. Impacts on Math Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  11. Impacts on Math Participating Schools improved by 8.4 points more than Districts (p<.001). Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  12. Impacts on Math Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  13. Impacts on Math Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  14. Impacts on Math Participating Schools improved by 8.4 points more than Districts (p<.001). That’s 14.7% more versus baselines. Participating Schools improved by 11.1 net percentage points (p<.001). That’s 18.9% better than baseline. Districts improved by 2.6 net percentage points (p<.01). That’s 4.2% better than baseline. Participating Schools experienced 4.2 TIMES greater improvement or Effect Size (p<.001). That’s a 4.5 Effect Size versus baselines (p<.001) Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  15. Impacts on Math Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: • 11.1net improvement for Schools (p<.001) • 8.4 more than their respective Districts (p<.001) • 18.9% better than their Yr. 1 baseline(p<.001) • 14.7% better than Districts vs. baselines (p<.001) • Effect Sizes: • 4.2 times greater improvementfor net growth vs. Districts (p<.001) • 4.5 times greater improvement for growth from baselines vs. Districts (p<.001) Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  16. Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  17. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  18. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Districts improved by 1.6 net percentage points (p<.01). Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  19. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  20. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 10.3 net percentage points (p<.001). Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  21. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  22. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 8.7 points more than Districts (p<.001). Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  23. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  24. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  25. Reading: Pct. of Students Proficient OR Advanced Net Pct. Change Statistical Statistical Year 1 Year 2 Change from Yr 1 Significance Significance Schools Mean 56.9 67.2 10.3 6.5 18.1% 7.2 p<.001 p<.001 StDev 22.4 22.1 Effect Size Effect Size Districts Mean 61.2 62.7 1.6 2.5% p<.01 p<.01 StDev 21.1 20.7 Net Difference 8.7 15.6% p<.001 p<.001 Level of p=0.079 p=0.063 p<.001 p<.001 Significance Impacts on Reading Participating Schools improved by 8.7 points more than Districts (p<.001). That’s 15.6% more versus baselines. Participating Schools improved by 10.3 net percentage points (p<.001). That’s 18.1% better than baseline. Districts improved by 1.6 net percentage points (p<.01). That’s 2.5% better than baseline. Participating Schools experienced 6.5 TIMES greater improvement or Effect Size (p<.001). That’s a 7.2Effect Size versus baselines (p<.001) Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013.

  26. Impacts on Reading Comparative Growth for Participating Schools In percentage of students Proficient or Advanced: • 10.3net improvement for Schools (p<.001) • 8.7 more than their respective Districts (p<.001) • 18.1% better than their Yr. 1 baseline(p<.001) • 15.6% better than Districts vs. baselines (p<.001) • Effect Sizes: • 6.5 times greater improvementfor net growth vs. Districts (p<.001) • 7.2 times greater improvement for growth from baselines vs. Districts (p<.001) Shaha SH, & Ellsworth H. Quasi-experimental Study of the Impact of On-demand Professional Development on Students Performance. International Journal of Evaluation and Research in Education (IJERE). Vol.2, no 4, Dec. 2013. *The original study of 15.3% cited has an errata underway to become 18.1%.

More Related