1 / 21

Impact of Interruptions on Test Scores in Indiana

Impact of Interruptions on Test Scores in Indiana. Richard Hill June 25, 2014. Two Parts. Initial study prior to presentation to Legislative committee Quick Outline Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf Follow-up study

konala
Download Presentation

Impact of Interruptions on Test Scores in Indiana

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Impact of Interruptions on Test Scores in Indiana Richard Hill June 25, 2014

  2. Two Parts • Initial study prior to presentation to Legislative committee • Quick Outline • Details in paper: http://www.nciea.org/publication_PDFs/ISTEP%20RH072713.pdf • Follow-up study • http://www.nciea.org/publication_PDFs/GainsMadebyInterruptedStudents_RH092513.pdf

  3. Three Confounding Factors • New policy for retention of students in Grade 3 • Transition from paper and pencil to computer administration • Interruptions

  4. Changes in Mean Scores over YearsELA

  5. Changes in Mean Scores over YearsMathematics

  6. Change from Paper and Pencil to Computer Administration • 2009 and 2010 – < 10% • 2011 – 36% • 2012 – 71% • 2013 – 95%

  7. Change Not Even Across Grades • 2012 • Grade 6 – 66% • Grade 7 – 86% • Grade 8 – 92%

  8. Additional Center Analyses • School-by-school improvement at same grade between 2012 and 2013 • School-by-school gain, following same cohort of students across grades • Student-level gain by students matched from 2012 to 2013

  9. CTB Analyses • Group Analyses • Overall statewide averages • Interrupted vs. non-interrupted within 2013 • Scores before interruption vs. scores after interruption • Individual Analyses • Before vs. after interruption • Performance predicted from previous tests

  10. Presentation to Legislative Committee • Was overall finding of no change a function of two factors? • Some students adversely affected by interruption • Other students taking advantage of interruption to learn answers from outside sources, then changing answers when testing restarted

  11. Model of Concern

  12. Available Data • CTB could provide A+B and C+D • That is, they knew at which item student was interrupted, and they knew, by item, whether student had made a change • But they couldn’t tell (easily) exactly the time the change was made

  13. Analysis of Table • Cell B is the event of interest—changes made after interruption to items presented before interruption • But C = 0, and A should equal D • So B = (A + B) – (C + D)

  14. Example—Grade 3 Mathematics, Session 1

  15. Results for Grades 3-5

  16. Results for Grades 6-8

  17. Conclusion • Concern raised by legislators was confirmed—data show that students changed answers from wrong to right more often after interruption than before • Impact on overall results was negligible—less than 0.2 scaled score points (on tests with standard deviations of 50-75)

  18. % Making No Changes from Wrong to Right (Grades 3-5, Math Only)

  19. Average % of Change from Wrong to Right (Grades 3-5, Math Only)

  20. Conclusions from Second Analysis • Students reported by CTB as interrupted had higher rates of change from wrong to right • Again, estimated impact is less than 0.2 scaled score points • Students reported as interrupted by locals had lower rates of change from wrong to right

  21. Consistent with first Impact on overall results negligible-- Conclusion from Second Analysis

More Related