In Pursuit of Testing Excellence

1. 2007 Annual Conference In Pursuit of Testing Excellence The Experts Respond To Your Practical Testing Questions

2. Moderator and Coordinator Chuck Friedman Consultant Council on Licensure, Enforcement and Regulation

3. Panelist Reid Klion Chief Science Officer pan-a TALX Company Council on Licensure, Enforcement and Regulation

4. Panelist Rose McCallin Director of Examination Services Colorado Department of Regulatory Agencies Council on Licensure, Enforcement and Regulation

5. Panelist Paul Naylor Consultant Council on Licensure, Enforcement and Regulation

6. Speakers YOU………the audience!!!! Council on Licensure, Enforcement and Regulation

7. Credentialing and Exam Issues at CLEAR • Challenges to Quality: Electronic and Candidate Integrity and Its Implications • Subject Matter Experts: Friends or Foes? • Getting the Most Out of Your Face-to-Face and Virtual Group Meetings • Get Ready for Continuous Computer- Based Testing: Examination Quality and Item Security Council on Licensure, Enforcement and Regulation

8. Credentialing and Exam Issues at CLEAR • Best Practice in Job Analysis and Validation of Test Specifications • Promoting Excellence in Managing ADA Accommodations • Quality Assurance and Regulatory Excellence • Vital Implementation Issues in Translation • Innovations and Trends in Testing Council on Licensure, Enforcement and Regulation

9. Responding to YOUR Testing Questions Council on Licensure, Enforcement and Regulation

10. Validity Reliability Item Writing Item Analysis Item Review Scoring Passing Score Standard Administration Test Blueprint Council on Licensure, Enforcement and Regulation Job Analysis

11. Council on Licensure, Enforcement and Regulation

12. Scoring Item Review Reliability Job Analysis Item Writing Validity Standard Administration Test Blueprint Item Analysis Passing Score Council on Licensure, Enforcement and Regulation

13. What are some alternative ways to conduct a job analysis? Job analysis Council on Licensure, Enforcement and Regulation

14. What are some alternative methods to conduct a job analysis? • Full Survey • Literature Search • In-Depth Interviews • Focus Groups Council on Licensure, Enforcement and Regulation

15. What are examples of some advanced, alternative item formats? What are the benefits and drawbacks of using these formats? Council on Licensure, Enforcement and Regulation

16. Innovative Item Types Options are Limitless….. • Performance Testing • Analogue Work Tasks - Visual Refraction - Medical Diagnostic Processes - Plant Operators • Streaming Video and Audio • On-Screen Manipulables Council on Licensure, Enforcement and Regulation

17. …..but Reality Constrains • Pragmatics • Usability • Logistics • Security ~ Salience of Rich Media • Development Costs Often High What are we getting for our money? • Psychometric Considerations Measures More Difficult to Develop than Methods Council on Licensure, Enforcement and Regulation

18. If the testing process is disrupted, (lost answer sheets/data, power failure, computer crash), what should my Board tell the candidate and what action should we take? Council on Licensure, Enforcement and Regulation

19. Stuff Happens… • Force majeure (“a greater force”) Tornado Strikes Brooklyn—August 2007 Council on Licensure, Enforcement and Regulation

20. What Could Happen? • Natural Disaster • Power/Internet Failure • Local Issue/Emergency • Many Events Unavoidable Can Only Plan to Mitigate Impact • Policy to Deal with Issues Council on Licensure, Enforcement and Regulation

21. Mitigation of Computer-Based Issues • Test Provider Issues - Redundant Systems • Backups • End-User (or Test Taker) Problems - Backups of Test Responses - Transmission to Servers After Each Screen Movement -Ability to Resume Test Session Council on Licensure, Enforcement and Regulation

22. What feedback should be provided to candidates on score reports and results? Council on Licensure, Enforcement and Regulation

23. What feedback should be provided to candidates on score reports and results? • Scoring Methods • Content Domain Scores • Interpretations • Reliability Issues • Cautions • Aggregate Scores • How to Reapply Council on Licensure, Enforcement and Regulation

24. Should my Board or program sponsor allow candidates to review test results including test questions? Council on Licensure, Enforcement and Regulation

25. AERA, APA, NCME Standard 8.2 Disclosure May Threaten Validity of Interpretations on Future Exams Candidate Is Entitled to Fair & Reasonable Appeals Process AERA, APA, NCME Standard 8.13 Allow Candidate Review? Council on Licensure, Enforcement and Regulation

26. AERA, APA, NCME Standard 1.10 Discourage Over-Interpretation of Information Subject to Considerable Error Candidate Interprets P/F Status on Basis of Isolated Items AERA, APA, NCME Standard 1.14 Allow Candidate Review? Council on Licensure, Enforcement and Regulation

27. Allow Candidate Review? • Candidate Right To Fair, Meaningful Appeals Process • Some Programs Allow Limited Access to Certain Test Items and At High Costs • Only Review M/C Items • Cost per M/C Item = $100+ • Process for Review Not Realistic • If Credit Granted, “Not Accredited” Council on Licensure, Enforcement and Regulation

28. Allow Candidate Review? • How Minimize Validity Threats of Exposure & Provide Candidates Meaningful Appeals Process? • Independent Review by Qualified 3rd Party • Non-Disclosure Contract • Supervised Environment • Authority to Challenge Council on Licensure, Enforcement and Regulation

29. What should I tell a candidate who failed the examination by one point? Council on Licensure, Enforcement and Regulation

30. Failed by “1 Point”? • Regulatory Exams: Aims of Measurement & Validity of Interpretations Maximize Accuracy of P/F Classification Decisions Council on Licensure, Enforcement and Regulation

31. Failed by “1 Point”? • Validity of Interpretations of Performance on Regulatory Exams • Maximize Consist, Accurate Classification Decisions • Consequences of Errors of Measurement • Candidates Whose Performance Is Far Above or Far Below P/F Decision Point • Candidates Whose Performance Is Close to P/F Decision Point (False Positives & False Negatives) Council on Licensure, Enforcement and Regulation

32. Failed by “1 Point”? Provide Results Reports that Encourage Interpretations Warranted by Precision of Classification Consistency Council on Licensure, Enforcement and Regulation

33. Failed by “1 Point”? Council on Licensure, Enforcement and Regulation

34. What are the practicalities and economics of maintaining a small licensing program? Quality Council on Licensure, Enforcement and Regulation

35. Strong As Weakest Lnk PIA & Result Post-Audits JA for Reg Prelimn. Try Outs Entry- Level Reqmts. Cut Scores Test Specs. Item Devel. Small Programs: Practicalities/Economics Council on Licensure, Enforcement and Regulation

36. Small Programs: Practicalities & Economics • Measurement Models: CTT or Rasch • Problems of Dimensionality • Carefully Consider Logic of Test Specifications, Content Dimensions • Best Practices • Good Practice Analysis, Test Specs. & Appropriate, Well-Formulated Test Items & Try-Outs Council on Licensure, Enforcement and Regulation

37. Validity As A Unified Concept Council on Licensure, Enforcement and Regulation

38. What are the benefits and drawbacks of computer based testing as opposed to paper and pencil delivery? Council on Licensure, Enforcement and Regulation

39. Computerized Assessment - Why? Convenience - Test Wherever There is a Computer - Logistics Vastly Simplified - Management of Errata - Possibility of Continuous Testing - Immediate and Accurate Scoring • “Hanging” Chads • OpScan Issues Council on Licensure, Enforcement and Regulation

40. Facilitation of Data Management • Data in Electronic Format • Web-Based Data = Immediate Availability • Centralized Management of Assessment Programs • Access to Array of Psychometric Data (e.g., response latencies, test taker activity) - Facilitate Integration with Other IT Systems Council on Licensure, Enforcement and Regulation

41. Candidate Perception • Candidate/Member Preference - Familiarity/Perceived Ease of Use - Demographic Issues • Organizational Appearance Council on Licensure, Enforcement and Regulation

42. Why Continue to Use Paper-Pencil? • Programs That Must Test Massive Numbers Concurrently in One Locality • Where Economics Dictate • Very Small Programs • Program Costs “Hidden” • “Immovable” Impediments Council on Licensure, Enforcement and Regulation

43. We are starting a new licensing program which does not allow grandfathering. What issues need to be considered? Council on Licensure, Enforcement and Regulation

44. We are starting a new licensing program which does not allow grandfathering. What issues need to be considered? • Security • SMEs • Limited Disclosure • Portfolio Review Council on Licensure, Enforcement and Regulation

45. What are the issues and potential pitfalls in adapting/translating our examination? See you soon! Council on Licensure, Enforcement and Regulation

46. What are the issues and potential pitfalls in adapting/translating our examination? • Adaptation • Process • Beyond Translation • Back Translation • Intuitively Valid Council on Licensure, Enforcement and Regulation

47. Adaptation/Translation • Translators • Language Skills • Cultural Skills • Subject Matter Knowledge • Trained in Item Writing/Review • Field Testing • Ron Hambleton/Stephan Sireci Council on Licensure, Enforcement and Regulation

48. What is the relationship between the examination process and reciprocity between states? Between countries? Council on Licensure, Enforcement and Regulation

49. Relationship Between Exam Process & Mobility • Reciprocity • Jurisdictions Agree to Recognize Each Other’s Practitioners without Require to Demonstrate Competence • License May or May Not Be Required from Other Jurisdiction Council on Licensure, Enforcement and Regulation

50. Relationship Between Exam Process & Mobility • Endorsement • Jurisdiction Recognizes License from Another Jurisdiction • Licensee Relieved of Full Burden of Obtaining License in New Jurisdiction Council on Licensure, Enforcement and Regulation