Changes in test Scores with Multiple Sittings of CanTEST

1. Changes in test Scores withMultiple Sittings of CanTEST Philip Nagy

2. Rationale Research Questions • Do test scores change on repeating the test? • Is change related to length of time between sittings? Test Development Questions • Can data from repeaters be used in test calibration for new form development? Context: Receptive Skills Official Languages and Bilingualism Institute

3. The Data Listening Tests: Six forms with 15 short and 25 long passage items Reading Tests: Seven forms with 15 skim-and-scan, 20 reading passage, and 25 cloze items The Sample: Mean first score of 3.6, compared to 4.3 for those who write only once Assumptions • Difficulty of forms is balanced across sittings (true) • Samples writing each form are equivalent (untested) Official Languages and Bilingualism Institute

4. Listening Results: Sitting 2 minus Sitting 1 (N=179) Official Languages and Bilingualism Institute

5. Listening Results, another look Official Languages and Bilingualism Institute

6. Listening Results Interpretation • How important is the improvement? • On average, 3.6 points needed out of 40 to improve one band • So, 2.6 points is about 75% of a band improvement Official Languages and Bilingualism Institute

7. Listening Results Interpretation • Can the data be used for test calibration? • The changes in average item difficulty are different for the subtests • .088 for short passages • .052 for long passages • The difference of .036 (.088 - .052) is about the same as the standard error of the difficulty indices • Listening data from repeaters should not be used for item calibration Official Languages and Bilingualism Institute

8. Changes in Listening by Length of Time between Sittings 1Difference significant, p=0.05 Those who repeat sooner do better than those who repeat later Official Languages and Bilingualism Institute

9. Reading Results: Sitting 2 minus Sitting 1 (N=284) Note: Reading Score is doubled to give a total out of 80 rather than 60. Official Languages and Bilingualism Institute

10. Reading Results, another look Official Languages and Bilingualism Institute

11. Reading Results Interpretation • How important is the improvement? • On average, 6.5 points needed (out of 80) to improve one band • So, 3.45 points is about 55% of a band improvement Official Languages and Bilingualism Institute

12. Reading Results Interpretation • Can the data be used for test calibration? • The changes in average item difficulty are different for the subtests • +0.072 for skim-and-scan • +0.050 for reading passages • +0.002 for cloze • The largest difference of .070 (.072 - .002) is two to three times larger than the standard error of the difficulty indices • Reading data from repeaters should not be used for item calibration Official Languages and Bilingualism Institute

13. Changes in Reading by Length of Time between Sittings 1Difference significant, p=0.05 Those who repeat later actually do worse than those who repeat sooner Official Languages and Bilingualism Institute

14. Conclusion • Listening: • 30% of sample do more poorly on 2nd sitting • Average gain is 75% of a band score • Differences in gains across item types vary by an item standard error • Reading • 40% of sample do more poorly on 2nd sitting • Average gain is 55% of a band score • Differences in gains across item types vary by 2-3 times an item standard error • Both • Those who rewrite within six months do better • Data from repeaters should not be used for item calibration Official Languages and Bilingualism Institute