Data Forensics: A Compare and Contrast Analysis of Multiple Methods

1. Data Forensics: A Compare and Contrast Analysis of Multiple Methods Christie Plackner

2. Outlier Score • Applied to most of the methods • Statistical probabilities were transformed into a score of 0 to 50 • 10 = statistically unusual

3. Erasure Analysis • Wrong-to-right (WR) erasure rate higher than expected from random events • The baseline for the erasure analysis is the state average • One sample t-test

4. Scale Score Changes • Scale score changes statistically higher or lower than the previous year • Cohort and Non-cohort • One sample t-test

5. Performance Level Changes • Large changes in proportion in performance levels across years • Cohort and Non-cohort • Log odds ratio • adjusted to accommodate small sample size • z test

6. Measurement Model Misfit • Performed better or worse than expected • Rasch residuals summed across operational items • Adjusted for unequal school sizes

7. Subject Regression • Large deviations from expected scores • Within year – reading and mathematics • Across year – cohort within a subject • One sample t-test

8. Modified Jacob and Levitt • Only method not resulting in a school receiving a score • Combination of two indicators: • unexpected test score fluctuations across years using a cohort of students, and • unexpected patterns in student answers • Modified application of Jacob and Levitt (2003) • 2 years of data • Sample size

9. Principal Component Analysis • Does each method contribute to the overall explained variance? • Can the methods be reduced for a more efficient approach?

10. Multiple Methods • Erasure Analysis (mER) • Scale score changes using non-cohort groups (mSS) • Scale score changes using cohort groups (mSC) • Performance level changes using non-cohort groups (mPL) • Performance level changes using cohort groups (mPLC) • Model misfit using Rasch Residuals (mRR) • Across subject regression using reading scores to predict mathematic scores (mRG) • Within subject regression using a cohort’s previous year score to predict current score (mCR) • Index 1 of the Modified Jacob and Levitt evaluating score changes (mMJL1) • Index 2 of the Modified Jacob and Levitt evaluating answer sheet patterns (mMJL2).

11. Principal Component Analysis • Grade 4 mathematics exam • 10 methods

12. Method Correlations

13. Principal Component Statistics

14. Scree Plot

15. Loading Matrix

16. Simplified Loading Matrix • +/- greater than 1/2 the maximum value in the component • (+)/(-) is between ¼ to ½ the maximum

17. Principal Component Statistics

18. Scree Plot

19. Reducing Variable Set • Determine how many components to retain • Cumulative percentage of total variation • Eigenvalues • The scree plot

20. Reducing Variable Set • Select one method to represent a component • Selecting methods within components • Positive selection • Retain highest loading method with components • Discarded principal components • Remove highest loading method with

21. Reducing Variable Set • Cohort regression* • Modified J&L, Index 1* • Non-cohort scale score change • Model misfit

22. Conclusion • All methods seem to account for variation in detecting test taking irregularities • Accounting for the most • Cohort regression • Cohort scale score change • Cohort performance level change • Method reduction results the same

23. Discussion • Different component selection methodologies • Closer examination of variables • Remove cohort regression or cohort scale score change • Combine the J&L indexes • Remove erasures