1 / 28

You’ve Been Shown: Now It’s Your Turn to Ask!

You’ve Been Shown: Now It’s Your Turn to Ask!. Responding to Your Questions About Testing. CLEAR 2004. Kansas City, Missouri. Credentialing and Exam Issues: Sessions at CLEAR 2004. Ten Year Review of CBT When Certification Replaces Licensure: Show me Accreditation

colm
Download Presentation

You’ve Been Shown: Now It’s Your Turn to Ask!

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. You’ve Been Shown: Now It’s Your Turn to Ask! Responding to Your Questions About Testing CLEAR 2004 Kansas City, Missouri

  2. Credentialing and Exam Issues: Sessions at CLEAR 2004 • Ten Year Review of CBT • When Certification Replaces Licensure: Show me Accreditation • ADA: Let Us Show You What Works • Show Me What Globalization Means for Credentialing Programs • Multiple Tracks to Licensure: Show me the Way • Test Piracy and Security in a Wired World CLEAR 2004 Kansas City, Missouri

  3. Credentialing and Exam Issues: Sessions at CLEAR 2004 • Show Me the Validity • Show Me How to Get Past MCQs: New Opportunities in Measurement • Show me an Evidential Approach to Assessment Design • The Case of the Repeating Licensure Examination Candidate: Psychometric, Legal, and Policy Issues CLEAR 2004 Kansas City, Missouri

  4. Moderator:Chuck Friedman Assistant Vice President, ACT Panelists: Carol Morrison Assistant Vice President National Board of Medical Examiners Elizabeth Witt Senior Statistician American Board of Emergency Medicine Rose McCallin Director of Office of Examination Services State of Colorado CLEAR 2004 Kansas City, Missouri

  5. Speakers You . . . the audience!!! CLEAR 2004 Kansas City, Missouri

  6. Validity Reliability Standard Administration Item Writing Item Analysis Item Review Scoring Passing Score Test Blueprint Job Analysis

  7. Scoring Item Review Reliability Job Analysis Item Writing Validity Standard Administration Test Blueprint Item Analysis Passing Score

  8. Should state licensing boards require continuing education? CLEAR 2004 Kansas City, Missouri

  9. How do I find out if my test has been compromised? If so, what should I do? CLEAR 2004 Kansas City, Missouri

  10. What should I do if I do or not do if I think a candidate or candidates have cheated? CLEAR 2004 Kansas City, Missouri

  11. Test Compromise? • Missing test book • When, who? • Statistical analyses • Memorization of items • Memorize and reconstruct items • Study guides • Internet postings • Statistical analyses

  12. Is a self assessment exam good enough for continuing competency? CLEAR 2004 Kansas City, Missouri

  13. Four Components ofContinuous Certification • Professional Standing • Lifelong Learning and Self Assessment • Assessment of Cognitive Expertise • Assessment of Practice Performance

  14. Considerations in Determining the Value of a Self-Assessment • How is the exam developed and scored? • Can you be certain who took the exam? • How high are the stakes for maintaining the credential? • Does the self-assessment process foster learning? • Is it easy to participate as part of one’s normal professional activities?

  15. Is performance testing useful in state licensure? CLEAR 2004 Kansas City, Missouri

  16. Performance Assessment • Does performance assessment measure something different? • Reliability, validity • How do you score it? • Cost, complexity • Answer: It depends!

  17. What is your opinion about open book exams? CLEAR 2004 Kansas City, Missouri

  18. What should I do if a candidate receives an incorrect score report? CLEAR 2004 Kansas City, Missouri

  19. How do you determine if a request for accommodations is appropriate? If a test is administered under special conditions, should the score report be “flagged”? CLEAR 2004 Kansas City, Missouri

  20. Is Request Appropriate? • Use expert consultant • Level the playing field, don’t tilt it too far the other way

  21. To Flag or Not to Flag? • Many testing companies have stopped flagging • Some claim that flagging may lead to unintended consequences • Standards for Educational and Psychological Testing (APA/AERA/NCME) • Flag if do not have evidence of score comparability

  22. Do we have to equate our tests? Is there one best way? ONE BEST WAY? CLEAR 2004 Kansas City, Missouri

  23. Reasons to Consider NOT Equating Small sample size Multidimensional exam Changing content outline

  24. What If We Can’t Equate? • Create a detailed blueprint • Write the best possible items • Build forms that are closely parallel in both content and statistics • Conduct frequent standard setting studies • Include common items across forms; study differences qualitatively

  25. An examinee complains that he failed the test by one point/question. What do you tell him? CLEAR 2004 Kansas City, Missouri

  26. Other Sources of Information • CLEAR Exam Review • Other CLEAR Publications • ADA Handbook • Development, Administration, Scoring and Reporting of Credentialing Exams • CLEAR web site: Frequently Asked Questions • Joint Standards: AERA, NCME, APA • Professional psychometricians CLEAR 2004 Kansas City, Missouri

More Related