1 / 49

The Case of the Repeating Licensure Examination Candidate: Psychometric, Policy, and Legal Issues

This presentation discusses the various reasons for retaking licensure exams, the impact on public protection, alternative forms of assessment, remediation options, concerns with retakes, and potential solutions to address fairness and protect the public. Presented at the 2004 CLEAR Annual Conference.

johnljames
Download Presentation

The Case of the Repeating Licensure Examination Candidate: Psychometric, Policy, and Legal Issues

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. The Case of the Repeating Licensure Examination Candidate:Psychometric, Policy, and Legal IssuesModeratorI. Leon Smith, President & CEOProfessional Examination ServicePresenters:Mary Browne, Program DirectorProfessional Examination ServiceJulia C. WorksAtkinson & Atkinson Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  2. The Case of the Repeating Licensure Examination Candidate: Psychometric and Policy IssuesMary Browne Program Director, PES Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  3. Why Retake? • Failure • Exam Malfunction • Suspicion of cheating • Illness or other incident Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  4. Second, Third…..Chances • Legal rulings support programs that allow retakes • The spirit behind most US education and law is to allow second chances • How does this impact on public protection • Should there be a cutoff on number of retakes allowed Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  5. Instead of a Retake • Alternative form of assessment • Partial retake - Retaking only domains or sections failed Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  6. Remediation • Should it be required • Is it effective Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  7. Retake Concerns • Overlap with previous forms • Timing • What results patterns are expected? Suspicious? • Practice effect • False Positives • Adverse Impact Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  8. Overlap With Previous Forms • Repeated Items • Equating • Item Bank limitations. • How do candidates perform on a repeated item? • Repeat previous response • Intervening learning based on recollection of the item? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  9. Retake Timing Considerations • How frequently is testing offered? • How many forms of the test are available? • How different are test forms? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  10. Reasons for Concern • False Positives • False Negatives • Errors of Measurement Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  11. True Score and Standard Error • Obtained score = True Score + Random Error 83 = 83 ± 3 or 80-86 • Errors are extraneous variables such as the fatigue, the particular sample of questions chosen for the test, environmental conditions or luck in guessing. • For every candidate there is a “best” and “worst” score possible. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  12. Some Examples of Score Changes in a Licensing Program. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  13. From One Exam to the Next Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  14. Two Common Failing Score Patterns Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  15. Score Patterns - Failing Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  16. Score Patterns - Failing Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  17. Passing Score Patterns Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  18. Score Patterns - Passing Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  19. Some Score History Illustrations Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  20. Passpoints • Most passpoints are set to pass the minimally qualified candidate. • Candidates whose qualifications are at a level just below the minimum may pass due to score error. • With repeated testing, this becomes more likely. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  21. Score Patterns - Passing Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  22. Dilemmas • Balance between fairness in allowing candidate retests and protecting the public from unqualified candidates. • Eliminating candidate advantage gained from repeated retakes Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  23. Remedies • Adaptive Testing • Set a Confidence Interval or Indifference Zone • Change the passpoint depending on number of retakes • Raise the passpoint to reflect retake advantage • Average last two or three test scores Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  24. Who Should Benefit? • Should decisions be weighted on the side of public protection or fairness to candidates? Where is the balance here? • Will more stringent retake requirements result in adverse impact? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  25. Contact Information Mary Browne PES 475 Riverside Drive 212 367-4250 (ph) 212 367-4266 (fax) browne@proexam.org http://www.proexam.org Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  26. Case of the Repeating Licensure Examination Candidate: A Legal PerspectiveJulia C. Works Atkinson & Atkinson1603 Orrington AvenueSuite 2080Evanston, IL 60201Tel. 847.864.0070Fax.847.864.0588E-mail: aanda@ameritech.net Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  27. Legal Perspective • Credential: • Voluntary • Private Sector vs. • High Stakes Licensure Examination: • Mandatory • Statutes and/or Rules • Public Protection Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  28. High Stakes Licensure Examination • Who relies on examination results? Regulatory Boards • For what purpose? • Measure of Minimum Competence • Protection of the public from unqualified practitioners Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  29. Legal Perspective • Limiting the number of attempts: How do regulatory boards look at this issue? • Security – exposure of exam items • Validity of Results • Does the fact that it took an applicant X number of times to pass the licensing exam mean they are less than minimally competent? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  30. Currently… • Majority of regulatory boards do not limit the number of examination attempts. • Those boards that do limit the number of attempts may require candidates to seek some type of remedial education before being permitted to sit for the examination again. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  31. Currently… • Many high stakes licensure examinations may not be retaken within a 90-day window • Examination attempts may be limited to a specified number within a specified time period • For example: • Limited to 3 attempts/retakes • Limited to 4 times within open window of administration of examination Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  32. How do you limit the number of attempts? • Statute • Rule/Regulation • Contractually • Policies and Procedures Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  33. Legality of Limitations • Constitutional Claims challenging limitation on number of attempts: • Equal Protection • Due Process Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  34. Equal Protection • Argument may be asserted that limiting the number of attempts violates Equal Protection because such limitations, whether established through statute or regulation, are notrationally related to any legitimate state interest. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  35. Linton v. Missouri Veterinary Medical Board • Legislature changed law to limit number of attempts to pass licensing exam to three(3). (Effective Aug. 1992) Facts: • Following revision to law, applicant failed licensing exam 3 times in MO but passed on the 4th attempt in Illinois • Applicant then applied to MO Board for license • Application was denied • Applicant appealed board decision asserting statute and regulation violated her right to Equal Protection Linton v. Missouri Veterinary Medical Board, 988 S.W.2d 513 (Mo.banc 1999). Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  36. Linton v. Missouri Veterinary Medical Board • Court held applicant’s right to Equal Protection not violated where three-exam limitation was rationally related to a legitimate state interest (i.e. the establishment of a high level of competence for veterinarians in Missouri). • Court found the interest of the legislature here was the same interest as establishing the test in the first place. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  37. Linton v. Missouri Veterinary Medical Board • Next question under Equal Protection Analysis: Is the means chosen rationally related to achieving the legitimate state interest? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  38. Linton v. Missouri Veterinary Medical Board In other words… • Are those who must take the exam four or more times before passing of equal or greater competence than those who pass the test in three or fewer attempts? • Is there a reasonable connection between competence and those who pass the exam in three or fewer attempts? Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  39. Linton v. Missouri Veterinary Medical Board • Following this decision… Statute amended to read, in part, as follows: …If that passing score was not received within three attempts, the board may require the applicant to appear before the board or submit evidence that the applicant has completed at least thirty hours of board-approved continuing education. V.A.M.S. 340.234 Examination– application and fees. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  40. Currently… • Similar restriction in Alabama with regard to the state examination where board rule reads, in part, as follows: (4) A maximum of three retakes of the state written exam will be permitted. Alabama Admin. Code 930-X-1-.09.Examinations Alabama State Board of Veterinary Medicine. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  41. Due Process • Allegations that limitation on number of attempts violates Due Process clause of U.S. Constitution because it deprives individuals of a vested right to take the licensing examination. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  42. State Board of Registration for the Healing Arts v. Boston Statute limited the number of times physical therapist assistants could take licensing examination Facts: - Applicant failed physical therapy assistant exam three times prior to August 1999, when statute limiting number of attempts became effective. - Applicant applied to take physical therapy assistant exam for 4th time after the statute effective date. - Board denied application. Administrative Hearing Commission reversed Board decision and ordered Board to admit her to the examination. - Board sought review of Administrative Hearing Commission decision that found statute could not be applied retroactively. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  43. State Board of Registration for the Healing Arts v. Boston • Court found that the legislative intent of the statute clearly manifested that it be applied retroactively. • Statute: • The board shall not issue a license to practice as a physical therapy assistant or allow any person to sit for the Missouri state board examination for physical therapist assistants who has failed three or more times any physical therapist assistant licensing examination administered in one or more states or territories of the United States or the District of Columbia V.M.S.A. Section 334.655.3, amended Aug. 28, 1999 Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  44. State Board of Registration for the Healing Arts v. Boston • Court found that the Missouri Constitution bars retrospective application of a statute except where: 1) legislative intent is clearly manifested that the statute is to be applied retrospectively; and 2) the statute is procedural only and does not affect any substantive or vested right. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  45. State Board of Registration for the Healing Arts v. Boston • Court found that statutory language unmistakably refers to past conduct • Court further held that application of the statute did not deprive applicant of any vested right. • As a general matter, professional licensing in a particular profession is a privilege granted by the state. • Restrictions surrounding the practice of a profession are for the benefit of society, not the practitioner • Licensing statutes confer no substantive rights upon applicants seeking licensure Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  46. Other situations to be considered • Licensure by Endorsement • See Yap v. Zollar, 691 N.E.2d 18 (Ill. App. 1 Dist. 1997) Statute requiring candidates for licensure as registered nurses to have passed nationwide exam within 3 years of first attempt rather than passing within 6 attempts within an unlimited period of time could only be applied prospectively because this was a substantive change and retroactive application would deprive applicant of a vested right where applicant had reasonably relied on prior statute. • See also John v. Dept. of Professional Regulation, 713 N.E.2d 673 (Ill. App. 1 Dist. 1999). Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  47. Other situations to be considered • Foreign Graduate Equivalency Programs Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  48. Contact Information Julia C. Works Atkinson & Atkinson1603 Orrington AvenueSuite 2080Evanston, IL 60201Tel.: 847.864.0070Fax: 847.864.0588E-mail: aanda@ameritech.net Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

  49. Reference Millman, J. (1989). If at First You Don’t Succeed: Setting Passing Scores When More Than One Attempt is Permitted. Educational Researcher, 18(6), 5 – 9. Presented at the 2004 CLEAR Annual Conference September 30 – October 2 Kansas City, Missouri

More Related