1 / 29

A Ten-Year Review of CBT

A Ten-Year Review of CBT. By Casey Marks, Ph.D., NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph.D., PV Presented at the 2004 CLEAR Annual Conference : 21 st Century Regulation: “Show Me” What Works September 30, 2004. Session Agenda. Speaker backgrounds

akio
Download Presentation

A Ten-Year Review of CBT

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. A Ten-Year Review of CBT By Casey Marks, Ph.D., NCSBN Geraldine Piskorski, MBA, ASCP Board of Registry Anthony R. Zara, Ph.D., PV Presented at the 2004 CLEAR Annual Conference: 21st Century Regulation: “Show Me” What Works September 30, 2004 2004 CLEAR Conference

  2. Session Agenda • Speaker backgrounds • Before transition to CBT • Transition • Operations • Refinement • CBT lessons learned • Conclusions 2004 CLEAR Conference

  3. Speaker Background Casey Marks, NCSBN Geraldine Piskorski, ASCP Board of Registry Anthony Zara, Pearson VUE 2004 CLEAR Conference

  4. What We’re Trying to Accomplish • Share experiences • Compare and contrast large vs. small programs’ issues • Share solutions • Outline successful practices 2004 CLEAR Conference

  5. Exploding numbers of programs More complicated item types Security concerns Globalization Maturing technologies, e.g., Internet Increased choices for program sponsors Some Industry Trends 2004 CLEAR Conference

  6. Small program ASCP/BOR background Size and depth of item bank Variable vs. fixed length test Opportunity to review Size of Examinee population Availability of computer testing facilities Cost of delivery Large program NCSBN background NCLEX facts Pre-CAT research program Barriers? Issues Before Transition to CBT 2004 CLEAR Conference

  7. Foundational Pre-CBT Research • Dimensionality • IRT calibration • Scale stability • Comparability of results • Early on, much psychometric research was necessary; now a bit less, but still need to validate findings with your program. • Mode of administration • What flavor of CBT? (CAT not right for everyone) 2004 CLEAR Conference

  8. Operational Research • Examinee interface • Stopping rules • Test length • Item pre-testing plan • Test timing • Candidate impact • Beta testing Here is where your program’s specifics get analyzed and determined 2004 CLEAR Conference

  9. Small program Transition from testing two days each year to year-round testing Staffing Stakeholder communications Large program CBT benefits Create roadmap to success Organizational processes Stakeholder communications Transition Issues 2004 CLEAR Conference

  10. Benefits of CBT • More frequent administrations • (daily testing) • Measurement accuracy and efficiency • Advanced item types can be used • Scheduling on demand • Increased security • Instant candidate results • Professional testing environment • Reduced time between retakes 2004 CLEAR Conference

  11. Roadmap for CBT • Needs analysis • Stakeholder analysis • Building consensus • Working decision processes • Communications • Leverage the experience of others when analyzing your program. 2004 CLEAR Conference

  12. Organizational Processes • Organizational Decision Process • Understanding needs • Options for solutions • Contracting Process • RFP or directed proposals • Provider options • Focus on outcomes 2004 CLEAR Conference

  13. Small program Psychometrics Administration Stakeholder support Large program Batch to continuous model Test development Psychometrics Administration Stakeholder support Staffing Operations Issues 2004 CLEAR Conference

  14. Program Operations • Batch to Continuous: Different Mindset • Test Development • Constant pool rotation -- ability to introduce content more quickly • More items needed • Continuous item development/reviewing • Item writer recruitment 2004 CLEAR Conference

  15. Program Operations • Batch to Continuous: • Psychometrics • Constant item maintenance • Different item exposure concerns • Candidate volume spikes/dips due to content changes and passing standard changes • Ability to remove underperforming problematic/underperforming items during administration 2004 CLEAR Conference

  16. Program Operations • Batch to Continuous: • Test Administration • Individual issues/incidents • Ongoing test center monitoring • Ratio of investigations to candidates increases • Time from exam registration to result • Time-dependent registration policies • Application process • Fee changes • Rescheduling issues 2004 CLEAR Conference

  17. Program Operations • Interface with stakeholders • Test accommodations • Communicating program changes • Procedure/documentation changes • Stakeholder Support • Direct support • Information/communication • Accessible • Timely • Constant • Rumors/urban legends 2004 CLEAR Conference

  18. Program Operations • Batch to Continuous: • Program Staffing • Dedicated • Specialized • Outsourcing • Customer Service • The on-demand mentality • Expectations 2004 CLEAR Conference

  19. Small program Immediate pass/fail results at test center Providers - more options Quality of graphics Large program Ongoing research Item types Providers Service CBT Program Refinement 2004 CLEAR Conference

  20. Program Refinement - Research • Ongoing Research • Methodology and content • Staff and/or test service • External • Implementation dates seemingly “arbitrary” 2004 CLEAR Conference

  21. Program Refinement – Item Types • Advanced item types • Media • Stimuli • Response formats 2004 CLEAR Conference

  22. Program Refinement - Providers • Service Provider Change • Between 1994 and 2002 same CBT provider • New test development and administration provider • (2 for 1) in 2002 • New center network • New test driver/candidate interface • Candidate support • Data reconciliation between test services • New application/administration/reporting/results processing systems • Seamless… 2004 CLEAR Conference

  23. Program Refinement - Service • Innovative item types (beyond MCQ) • Utilizing same exam administration parameters • More frequent administrations • 2 to 4 to 8 per year • Additional testing time • Opportunity to demonstrate competence under optimal exam termination rules • International administration • Outside U.S. and territories starting in 2005 2004 CLEAR Conference

  24. Small program Importance of good communication with all stakeholders Size of examinee population Improved benefits to examinees Large program Benefits obtained worth it? Program complexities Stakeholder satisfaction? Service? Etc. CBT Lessons Learned 2004 CLEAR Conference

  25. Challenges • Large program, yet perfection is needed/expected daily… • More moving pieces, more expertise needed • Change is harder to integrate • The more complex the program, the more difficult to educate stakeholders • Timing of program changes never convenient for all stakeholders 2004 CLEAR Conference

  26. General Conclusions • Any large vs. small program things? • Any “gotchas”? • Need to consider all program stakeholders when considering procedure changes 2004 CLEAR Conference

  27. Conclusions • CAT is a good thing for the NCLEX… • Boards of Nursing enjoy: • Convenience and speed of system • The ability to focus on exam content not psychometric and/or exam administration • Candidates like: • Quick registration, scheduling and results receipt • Scheduling options • Personalized service • NCSBN appreciates: • Security benefits • The ability to test content in new and novel ways 2004 CLEAR Conference

  28. Discussion 2004 CLEAR Conference

  29. Casey Marks, Ph.D. Director of Testing Services NCSBN email: cmarks@ncsbn.org Gerry Piskorski, MBA Director, Applicant, Certificant and IT Activities ASCP email: gpiskorski@ascp.org Anthony R. Zara, Ph.D. Vice President Testing Services Pearson VUE email: tony.zara@pearson.com Contact Information 2004 CLEAR Conference

More Related