ORAL CASE EXAMINATIONS FOR ASSESSING INTERN COMPETENCE

1. ORAL CASE EXAMINATIONS FOR ASSESSING INTERN COMPETENCE Robt. W. Goldberg, Ph.D., ABPP Kevin R. Young, Ph.D. Louis Stokes Cleveland DVA Medical Center

2. Learning Objectives • 1. Learn the rationale for an intern final oral competency examination • 2. Acquire knowledge of the relationships between orally assessed and supervisor rated competencies • 3. Understand some implications of the findings for models of trainee competence

3. Background: • Graduate school qualifying exams generally test book knowledge. • Dissertation oral examinations test application of scientific knowledge to a particular research problem. • Licensing exams typically test knowledge of a particular state’s laws and regulations, with perhaps a vignette to examine application of APA Ethical Principles and Standards Conclusion: Nowhere is it required that clinical competence be formally tested with respect to practical, clinical experiential application. Implication: It would be desirable to do so before the attainment of licensure for independent practice. With the advent of Ohio licensure possible upon doctoral program graduation, without further postdoctoral supervision, our program decided to develop an Intern Final Oral Examination of clinical competence.

4. LSCDVAMC Prior Research: We compared four different models of competence in oral examinations of our 2009-2010 ten-intern cohort. These models were: 1. California School of Professional Psychology (Dr. Patrick Petti). 2. APA Competency Benchmarks (2009 version) 3. American Board of Professional Psychology Clinical Psychology Competencies 4. Our local system, dimensions derived from supervisor rating forms Conclusion: The CSPP model yielded the widest range of scores within an acceptable/passing range, was regarded as the most useful for providing feedback, and was the most preferred by both interns and examiners. Outcome: We adopted the CSPP model examination for use in our program.

5. CSPP Model: • Examines trainees on 6 Domains of competence: 1. Assessment 2. Formulation 3. Intervention strategy 4. Relationship 5. Self-examination 6. Professional communication skills • Each Domain has 4 or 5 Competencies per Domain, rated on 6 point scale, with an overall Domain summary score (not necessarily the arithmetic average) 1. Significantly below expectations 2. Below expectations 3. Marginally meets expectations 4. Clearly meets expectations 5. Exceeds expectations 6. Greatly exceeds expectations

6. Examination process: • CSPP uses their oral examination as a qualifier for applying to internships, and examines on a written work sample. We examined on an actual current case, presented orally to two trained faculty examiners, utilizing a , predetermined outline. Exam conducted at mid-point of the final four-month rotation, giving intern opportunity to remediate , and be re-examined if s/he fails. • Transparency: Outline for the case, CSPP criteria, forms used by the examiners, and background readings provided well ahead of time. The case has been supervised by staff and perhaps discussed in seminars and group supervision. 3. Rationale: Creating exam conditions which maximally permit intern competencies to emerge using a , familiar case, in a moderately realistic analog of a staff meeting. Examination format: • 30-minute uninterrupted oral presentation of the case • 45 minutes of examination • 15 minutes of examiner discussion and formal rating with intern not present • 15 minutes (or more) of feedback and scores to the intern Subjects: 38 interns in 4 internship years who successfully completed the internship.

7. Supervisor Training Evaluations: * Four broad areas of intern competence 1. Professional qualities and role behaviors 2. Clinical decision making 3. Assessment 4. Intervention * Each broad area of competence has 7 to 10 more specific constituent competencies (e.g. for the “Assessment” area, “Psychological testing – self report inventories” is a specific competency) * Supervisor ratings are made on a four-point scale of: 1. Remedial 2. Needs regular supervision. 3. Needs occasional supervision 4. Ready for autonomous practice * Overall Rating of Clinical Competence as a summative rating is also made on the 4-point scale

8. Data Analysis Exam: • Average Exam score was 50.8 (N = 38), SD = 7.7 • Relationship, Self-Examination, and Professional Communication Skills Domain scores were combined into one Professional Behavior score for comparability with supervisor Area ratings. • Average Domain scores were as follows (range = 1 – 6): Assessment = 4.15 Intervention = 4.18 Formulation = 4.34 Professional Behavior = 4.06

9. Supervisor ratings: • Ceiling effect: Area means were 3.48 – 3.71, all w/in .75 SD of maximum • Therefore: 1. Retrospective Supervisor Ratings were calculated based on average deviation of that supervisor’s rating from the mean. 2. Supervisors were asked, post hoc, to rate past supervisees on “perception of skill level” from 0 (“worst intern imaginable”) to 100 (“best intern imaginable”). • Reliability: Agreement between supervisors on the same intern’s Area ratings was very poor (Assessment [.38]and Intervention [.40]) to mediocre (Clinical Decision Making [.68] and Professional Behaviors [.65]) • Supervisors were consistent within their own frame of reference (alpha = .80 or greater)

10. * Interrater reliability was significantly better for examiners than for supervisors. * Why? Examiners observed the intern only during the oral exam, rating the same sample of intern behavior. Supervisors were rating different samples of intern behavior on the different rotations. • UnweightedKappas (which take all disagreements as equal) reflect that the examiners correlated .52 better than chance (i.e. had 2 raters randomly rated an examinee between 1 - 6 on each respective dimension. • Weighted Kappas (accounting for magnitude of disagreements between raters) reflected that the examiners correlated .84 better than chance (i.e. had 2 raters randomly rated an examinee between 1 – 6on each respective dimension.

11. RESULTS RESTROSPECTIVE SUPERVISOR RATING AREAS Overall Assessment Intervention CDM PB EXAM DOMAINS Total score .64 .24 .27 .35 .57 Assessment .29 .09 .11 .17 .40 Intervention .33 .16 .10 .21 .35 Formulation .33 .30 .30 .43.60 PB .31 .21 .35 .25 .58 p (.05) italicized p (.01) bold italicized

12. AAPIVariablesRelatedtoOralExamTotalScore Mean (n = 38) Correlation w/Exam Assessment Hours 223 .03 Intervention Hours 749 -.22 Nr of Integrated Reports 26 -.11 Nr of Articles 1.6 .24 Nr. of Chapters 0.3 -.03 Nr. of Presentations 5.5 .11

13. Interpretive Conclusions • Results from Intervention and Assessment Exam Domains did not show practically significant relationships with Supervisor-rated performance, whether cotemporaneously or retrospectively. • Formulation/Decision Making and Professional Behavior Domains did show significant relationships with Supervisor retrospectively rated performance in these Areas. • Selected AAPI variables did not predict Exam performance. • Interpretation: An overarching general construct or factor- the ‘clinicalg’- is present underlying intern competence: the ability to formulate and effectively communicate case conceptualizations. We speculate that this reflects a combination of mental capacity and the relevant psychology knowledgebase. • Limitations: (1) Different meanings and indicia of “Assessment” and “Intervention” for different supervisors, on different rotations, and in the Exam. (2) Exam Domains not effectively measuring the respective Supervisor-rated Areas. (3) A ‘halo effect’ of supervisors rating particular interns (all Areas high or low)

14. IMPLICATION Discriminability of discrete competencies in different competency models may be illusory. One broad factor, a ‘clinical g,’ may be the most important constituent of all specific competencies.