“Problems and Priorities in Pretesting Pondered”

1. “Problems and Priorities in Pretesting Pondered” Beth Noeller Grady Barnhill Carol O’Byrne

2. Pretesting – why all the fuss? • With today’s emphasis on computer-based testing – many programs provide instant results to candidates • Item analysis often done once/year on item bank • Generally not an opportunity for review of item analysis, comments and scoring changes CLEAR 2008 Annual Conference Anchorage, Alaska

3. Pretesting – why all the fuss? • Thus, much greater pressure for items to be right (therefore more need for pretesting) • Larger, more sophisticated programs have subroutines to control item exposure to prevent overutilization CLEAR 2008 Annual Conference Anchorage, Alaska

4. So what if there’s a small n? • Combine data from multiple administrations (but be careful of “data expiration dates”) • ….and?… CLEAR 2008 Annual Conference Anchorage, Alaska

5. Small n Item Analysis • In interpreting item analysis for small n… • Focus primarily on p value • Largely ignore discrimination – often spurious • Can look at pattern of response, if dramatic • Rely much more on SMEs. • Review more items than you would normally (set review parameters more conservatively than you would normally) CLEAR 2008 Annual Conference Anchorage, Alaska

6. …So….. What else? • So now we may have to get creative • Sometimes…. Something is better than nothing… but not always…. CLEAR 2008 Annual Conference Anchorage, Alaska

7. Multiple exams-common items • Although rare, some exams with multiple levels (Class 1, 2 & 3), use common items. • Combine data for common items from different classes. • Beware of violations of assumptions – is pretest meaningful? CLEAR 2008 Annual Conference Anchorage, Alaska

8. Multiple item reviews… • In a way – reviews by SMEs can considered pretesting • Especially important that reviewers not be original writers CLEAR 2008 Annual Conference Anchorage, Alaska

9. Cut Score Studies 1 • Use Ebel or Nedelsky… • Estimating probability of response for each choice provides critical review • Also simulates responses from candidates • Potential problem: difference in ability levels of SMEs and candidates CLEAR 2008 Annual Conference Anchorage, Alaska

10. Cut Score Studies 2 • When doing an Angoff – have SMEs take test 1st • Data gathered can be used as pretest information • (and also useful to use as a limit on standard setting) CLEAR 2008 Annual Conference Anchorage, Alaska

11. “Floating” pretest items • Don’t delineate items as pretest initially • Use item analysis to identify most flawed items • Declare worst performing items to be Pretest • Can be continued through life of examination • Care must be taken to still meet blueprint CLEAR 2008 Annual Conference Anchorage, Alaska

12. Specialty Exams • PMPs – Patient Management Problems • Complicated – essentially redeveloped with each deployment • Multiple item reviews/edits, help preserve quality • Secure, online review and conferencing facilitates review CLEAR 2008 Annual Conference Anchorage, Alaska

13. Sooooooooo…. ---Good Luck with that!!! CLEAR 2008 Annual Conference Anchorage, Alaska