1 / 38

ABA/ASA JOINT COUNCIL ON IN-TRAINING EXAMINATIONS

ABA/ASA JOINT COUNCIL ON IN-TRAINING EXAMINATIONS. 2005 REPORT TO SAAC/AAPD Glenn P. Gravlee, MD Chair, Joint Council [The Ohio State University Medical Center Department of Anesthesiology] Gravlee.1@osu.edu. ABA David Chestnut Douglas Coursin Glenn Gravlee Patricia Kapur

kenyon-bray
Download Presentation

ABA/ASA JOINT COUNCIL ON IN-TRAINING EXAMINATIONS

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. ABA/ASA JOINT COUNCIL ON IN-TRAINING EXAMINATIONS 2005 REPORT TO SAAC/AAPD Glenn P. Gravlee, MD Chair, Joint Council [The Ohio State University Medical Center Department of Anesthesiology] Gravlee.1@osu.edu

  2. ABA David Chestnut Douglas Coursin Glenn Gravlee Patricia Kapur - Jim Di Nardo Raymond Roy Kenneth Tuman ASA Arnold Berry John Rowlingson Jeffrey Gross (V.C.) Philip Lebowitz Charles Otto Patricia Petrozza Mark Rosen (Robert Gaiser) 2005 JOINT COUNCIL MEMBERS

  3. IN-TRAINING EXAMINATION • “3 exams in 1” • In-Training Exam • Requalifying Exam • ABA Written Exam • Subset of 300 questions from ITE • Plus: Recertification Exam • Well-performing A-types from ITE form the question pool

  4. Demographics of In-Training Examinations Total Number of Examinees Registrants

  5. Test Center Issues • Tulane and Univ of S. Alabama cancelled due to storm (not Katrina) • Two alternate dates offered; most took exam on July 30

  6. DEMOGRAPHICS OF IN-TRAINING EXAMINATIONS - AMG + IMG Number of Residents (n)

  7. Exam Content 2005340 Questions ~ 100 are Links 19% 13% 4% 8% 10% 45% Number of Questions

  8. Disease States Breakdown163 Questions Number of Questions 7.2% 5.7% 5.7% 5.1% 4.9% 4.6% 4.6% 4.6% 4.3%

  9. 2005 ITE Question Distribution Percent of Questions (%)

  10. Item Difficulty (Mean Logit ± SE)

  11. In-training ExamAverage Scaled Score Scaled Score All CA3 Residents

  12. Gaps in Knowledge – CA3s • 10 Surprising Gaps Identified • Report sent before ASA

  13. % Correctly Answered 2005 ITECalibration Group Percent Correct Answers (%)

  14. Survey of Program Directors • Sent to all residency programs in early 2005 • 95% of programs responded by 5/26/2005

  15. Joint Council 2005 Survey of Residency Programs Question: Assuming that the In-training Examination will be administered once a year, I would prefer that the exam be given in: Percent (%)

  16. Joint Council 2005 Survey of Residency Programs Question: The importance of having the in-training Examination be equivalent in content to the ABA written examination is: Percent (%)

  17. Joint Council 2005 Survey of Residency Programs Question: Having the opportunity to administer the In-training Examination in one, 3.5-hr, test session instead of two, 3.5-hr test sessions would be: Percent (%) Rationale: May need to shorten for computerized format

  18. Joint Council 2005 Survey of Residency Programs Question: Having the opportunity to administer the In-training Examination twice a year would be: Percent (%)

  19. Joint Council 2005 Survey of Residency Programs If the In-training examination should be administered twice a year, it would be best to: • Have all residents take it both times – 44% • Limit access on the second administration to residents with scores below a particular level (e.g. 20th percentile of their cohort) – 34% • Have the second administration be optional but available to all residents – 19% • Don’t do it – 1%

  20. Joint Council 2005 Survey of Residency Programs The ideal way for residents to take the In-training examination is: • On the same day at the same time – 54% • During an eight-day testing window beginning and ending on a Saturday – 46%

  21. Joint Council 2005 Survey of Residency Programs The ideal way to deliver the In-training examination to residents is via: • Printed test books – 27% • Computer at a commercial computer-based testing center – 21% • Internet to secure, proctored centers, which may be at the training institution – 45% • Both B & C – 2% • No answer – 5%

  22. Internet Based Exam • ABA survey of directors and Joint Council members strongly suggests that internet-based secure exam capacity of most teaching institutions remains quite low • Can’t determine that anyone has yet done an internet-based exam on this scale • AB Derm does it on a much smaller scale, and implementation was UGLY in 1st year

  23. Future of ITE • Computerized Exams target date: 2008 • Logistic Issues: • Cost (seat fees, capacity limitations) • Leads to separation of dates of ABA, ITE • Leads to security concerns • Requires larger question pool and larger number of exam “forms” while keeping content equivalent: drives development cost up

  24. Future of ITE • Computerized Exam advantages • Images more easily used and updated • Fewer concerns about proctor consistency (?), enforcement of rules (cell phones/pagers, cheating) • Facilitates interactive questions (expensive to develop) • No exam booklets to print, ship, track, or lose Bottom Line: We want to do it, but cost and other issues may hold us back. Dec meeting in Phil.

  25. Future of ITE • Consideration of grading or peer comparison for different subject areas, e.g. • Physiology, Pharmacology, Anes Procedures Cardiovascular, Respiratory, OB anesthesia • Section needs to have a critical mass of questions to achieve reliability • (Some questions will fall into more than one category)

  26. ABA Written Exam Failure Rate Failure Rate (%)

  27. 2005 ABA WRITTEN EXAMINATION PASS RATES1st Time Takers

  28. Pass Rates for 1st time takers of 2005 ABA Written Exam Grad Class = 2003 or 2004)

  29. Joint Council UpdateSummary • Exam population still growing: Max now? • 2005 resident exam performance: EXCELLENT • Computerized Exam: On the way (?) • Taking the ABA exam ASAP appears better than delaying it

  30. Examination Registration/Behavior • Government Issue Picture ID • NO Cell Phones or Pagers • Calculators are NOT OK • Consequences of Cheating • Proctors are in charge

  31. Joint Council Finances • Joint Oversight by ASA/ABA • Largely sustained by Exam fees • In the Red ($50-100K) when N was low

  32. Joint Council Finances • Still losing $20-50K/year with strong N: increased proctor compensation, NBME fees mainly • Exam cost @ $85 since 1996 • Joint Council approved increase in exam fee to $100 effective for 2005 exam

  33. Content Outline Revision • 1996 version used for 2004 Exam • 2003 version used for 2005 Exam • New Format: “Organocentric” • Both versions on ASA website • 2004 version disappears 12/31/04 • www.asahq.org/publicationsandservices/contentoutlinerev2003

  34. FAQs/Future Issues • Computerized Exam? • Phase out K-types? • Introduce R-types and question clusters (G types) in 2005

  35. 50 Question Writers Title: Junior Editors

  36. IN-TRAINING EXAMINATION EDITORS Steven Allen, Audree Bendo, James DiNardo, Carter Dodge, Sylvia Dolinski, John Ebert, John Emhardt, Robert Gaiser, Eric Kitain, Bruce Kleinman, Elliott Krane, Larry Krenis, Larry Kushins, Cynthia Lien, Vinod Malhotra, Donald Martin, Roger Mecca, John Moyers, Julia Pollock, Lindo Jo Rice, Robert Sladen, Richard Stypula, Richard Teplick, Helen Westman, Thomas Wolfe

More Related