“Putting a Luster on a Good Testing Program”

“Putting a Luster on a Good Testing Program” Barbara Showers, PhD Wisconsin Department of Regulation and Licensing Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Low-Tech Indicators of Potential Exam Problems • Frequent Score Adjustments • Pass Rate Changes • “I Can’t Find it on Your Web Site” Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Frequent Score AdjustmentsIndicators • Regular grade adjustments on practical or oral exams before release or on appeal • Increasing numbers of multiple-keyed or omitted multiple choice questions Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Frequent Score AdjustmentsPractical/Oral Issues • Are examiners unclear on the criteria? • Are examiners given too much flexibility? • Are the criteria themselves solid (one correct answer)? Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Frequent Score AdjustmentsPractical/Oral Solutions • Clarify scoring criteria • Reduce flexibility – checklists • Develop a bank of pre-approved questions with correct answers • Train new examiners/board members • Pair new examiners with experienced ones • Use multiple examiners Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Frequent Score AdjustmentsMultiple Choice Issues • Questions are out of date • Questions are ambiguous Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Frequent Score AdjustmentsMultiple Choice Solutions • Multiple question sources and writers • Committee review before use • Update on a regular schedule • Allow candidate comments after exam • If sufficient volume, run item analysis statistics – ID problems • Don’t keep the multi-keyed or omits in the test – reflects poorly on quality Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangesIndicators • Gradual pass rate increase over time, or • Pass rates swing up or down in short time Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangesIssues Reasonable Explanations for Pass Rate Change Exam content change Exam cut score change Candidate qualifications change Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangesIssues If no reasonable explanation, consider • Security breach • Standard setting issues – are the forms equivalent in difficulty? Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangeIssues • What is a “reasonable” pass rate? Example: WI Hearing Instrument Specialist Exam Audiologist Pass Rate = 70% Apprentice Pass Rate = 38% Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangesSolutions Security solutions • Multiple forms • New content as often as possible • Avoid repeating highly memorable scenarios Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Pass Rate ChangesSolutions Unexplained Variations • Equate forms if possible • If statistical equating is not possible, use at least subject matter expert ratings of difficulty to create similar forms • Define and document the passing performance standard and use the definition over time for consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

“I can’t find it on your web site.” • Put the luster on your testing program by polishing your web site Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Polishing the Web Site • Is the candidate bulletin prominent? • How deeply buried is the fee information? Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Polishing the Web Site • How to apply • What to expect in content and administration • Study resources • Links to other relevant sites Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

Speaker Contact Information Barbara Showers, PhD Wisconsin Department of Regulation and Licensing P.O. Box 8935 Madison, WI 53708 608-266-7703, fax 608-267-1809 barbara.showers@drl.state.wi.us http://drl.wi.gov Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona

“Putting a Luster on a Good Testing Program”