ACGME & OSCE Use of OSCE in Residency Evaluations

1. ACGME & OSCEUse of OSCE in Residency Evaluations Paul Gordon, MD EOSG May 2003

2. ACGME Toolbox • OSCE is listed as the most desirable for • patient care • interpersonal and communication skills • professionalism • next best method for • practice-based learning and improvement • systems-based practice • The toolbox suggests written and oral exams as the most desirable for assessing medical knowledge.

3. But… • OSCE measures have not traditionally been designed to take into account levels of expertise • creating better OSCEs require understanding of how interviewing style of experts differs objectively from novices

4. Continued Problems • OSCEs have traditionally employed binary content checklists to score behaviors demonstrated by students • growing recognition of the inability of checklists to capture complex human behaviors such as empathy and organization has resulted in the adoption of multipoint global ratings designed to capture non-binary characteristics of the performance

5. Students vs. Residents • professionals pass through five stages: • novice, advanced beginner, competence, proficiency and expertise • early novice stage is characterized by the collection of large amounts of data, in no particular order, with little regard for situational factors, which are then used to synthesize a problem solution

6. Students vs. Residents • Experts: • gather much more focused information • rely on many different types of data including situational variables • compare all observations to previous experiences and; • quickly and automatically respond to such observations, often without resorting to any formal process of problem solving

7. Students vs. Residents • experts cannot easily break down their thinking into component steps and therefore have great difficulty returning to a novice form of solving problems • checklists penalize clinicians who arrive at diagnoses quickly as a result of pattern recognition and rapid hypothesis testing

8. What to Do? • need to look at variables such as types and sequence of questions and statements • measures that are sensitive to the nature of expertise should include the sequence and organization rather than just the number of questions asked

9. What to Do? • while there appear to be patterns of interviewing that characterize different levels of expertise, they are subtle and may not be amenable to counting and classification for examination purposes

10. Interviewer Utterances • questions • summary statements • empathic comments • articulated transitions • information giving

11. Research • Mean number of utterances exceeded one every 10 seconds for all groups • largest proportion was Questions • 76% for clerks; 67% for experts • 1/3 of total utterances consisted of a group of “low frequency” types: • empathic comments, information giving and summary statements

12. Research • While utterance type over time appeared to show characteristic patterns reflective of expertise, the differences were not robust • only the pattern of use of Summary Statements was statistically different between groups (P<0.05)

13. Summary • Residents are different than students • Measures that are sensitive to the nature of expertise should be used to supplement OSCE checklists that simply count questions • Information giving, empathic comments and summary statements should be credited • While there appear to be patterns of utterances that characterize levels of expertise, the patterns are subtle and not always amenable to counting