Assessing Knowledge and Performance

1. Assessing Knowledge and Performance John Littlefield University of Texas Health Science Center at San Antonio

2. Recall a medical student or resident whose performance made you uneasy. • What behavior or event made you uneasy? • What action did you take? a. Talk with faculty colleagues about your concerns b. Write a candid performance appraisal and send it to the course/residency director 3. If you wrote a candid appraisal, did an administrative actionoccur related to the student/ resident?

3. Goals: Assessment of Knowledge and Performance • Clarify two distinct uses for assessments of clinical knowledge and performance • Define two aspects of validity for all assessment methods • Compare and contrast 6 techniques for assessing clinical knowledge and performance • Identify poorly written multiple choice test items • Write a key features test item • Describe a role for narrative comments in scoring interactions with Standardized Patients • Describe three elements of a clinical performance assessment system • Critique a clinical performance assessment system that you use

4. Agenda: Assessment of Knowledge and Performance • Exercise: Recall a medical student or resident whose performance made you uneasy • Presentation: Quality assurance when assessing clinical knowledge and performance • Exercise: Take then critique a multiple choice test • Presentation: Key features test items • Exercise: Write a key features test item • Exercise: Critique a videotaped student-SP interaction • Presentation: Widening the lens on SP assessment • Exercise: Recommend program director actions based on faculty comments about a resident • Presentation: Improving clinical performance assessment systems • Exercise: Critique your clinical performance assessment system

5. Uses for Assessment: Formative vs. Summative Purpose Feedback for Certification/Grading Learning Breadth of Narrow Focus on Broad Focus on Scope Specific Objectives General Goals Scoring Explicit Feedback Overall Performance Learner Affective Little Anxiety Moderate to High Response Anxiety Target Audience Learner Society

6. Validity of Knowledge and Performance Assessments • Content - Does the assessment method measure a representative cross-section of student competencies? • Structural – Degree to which content and scoring represent competence in each subsection of the performance domain • External - Do scores from this assessment method correlate highly with scores from other measures of the same student competencies? • Consequential - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment? • Generalizability • Does the student perform at about the same level across 5 to 7 different patients / case problems? • Does the student receive a similar rating from different faculty? • Substantive – the context surrounding the assessment evokes the domain of cognitive processes used by a physician

7. Six Aspects of Assessment Validity Viewed as a Cube Generalizability Consequential Content External Structural Substantive

8. Generalizability of Physician Performance on Multiple Patients

9. Validity of Knowledge and Performance Assessments • Content - Does the assessment method measure a representative cross-section of student competencies? • External - Do scores from this assessment method correlate highly with scores from other measures of the same student competencies? • Consequential - Do various subgroups of students (e.g., different ethnic groups) score equally well on the assessment? • Generalizability • Does the student perform at about the same level across 5 to 7 different patients / case problems? • Does the student receive a similar rating from different faculty? • Substantive – the context surrounding the assessment evokes the domain of cognitive processes used by a physician

10. Substantive Aspect of Validity: Four Levels of Performance Assessment 1 Does (Global Rating) Shows How ( OSCE) Knows How (Examination – Oral) Knows (Examination – Multiple-choice) • Miller, GE. Assessment of clinical skills/competence/performance, Academic Medicine, 65(9), supplement, S63-7, 1990

11. Comparisons among Six Techniques for Assessment of Clinical Knowledge and Performance________________________________________________________________________________________ Multiple Modified Structured Multimedia Standardized On-the- Choice Essay Oral Exam Simulation Patients Job PA • Knowledge +++ ++ + + + + • Interviewing/Interpersonal __ __ + + ++ + • Data gathering / history + ++ ++ ++ ++ + • Physical exam (technique) __ __ __ + +++ + • Reasoning / diagnosis + ++ ++ ++ ++ + • Lab utilization/management + ++ ++ ++ ++ + • Personal qualities __ __ __ __ __ ++___________________________________________________________________________________ + = adequate ++ = good +++ = excellent __ = not applicable Newble D, (1992). Assessing clinical comp. at undergrad level. Med. Educ. V. 26, 504-511

12. Interim Summary of Session • Session thus far • Two uses of knowledge and performance assessments: Formative and Summative • Validity of all assessment techniques • Comparisons among 6 assessment techniques • Coming up • Take and critique a 14 item multiple choice exam • Presentation on Key Features items

13. Knowing and Realizing • The mere knowledge of a fact is pale; but when you come to realize your fact, it takes on color. It is all the difference between hearing of a man being stabbed in the heart, and seeing it done. Mark Twain, A Connecticutt Yankee in King Arthur’s Court

14. How are Multiple Choice Items Selected for an Exam?

15. Sample Exam Blueprint based on Clinical Problems Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.

16. Key Features of a Clinical Problem • Definition: Critical steps that must be taken to identify and manage a patient’s problem • focuses on a step in which examinees are likely to make an error • is a difficult aspect in identifying and managing the problem • Example: For a pregnant woman experiencing third-trimester bleeding with no abdominal pain, the physician should: • generate placenta previa as the leading diagnosis • avoid performing a pelvic examination (may cause bleeding) • avoid discharging from clinic or emergency room • order coagulation tests and cross-match Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.

17. Test Items based on a Clinical Problem and its Key Features Page G, Bordage G, Allen T. Developing key-feature problems and examinations to assess clinical decision-making skills, Acad. Med. 70(3), 1995.

18. Scoring the Placenta Previa Clinical Problem • Key Feature 1: To receive one point, must list placenta previa or one of the following synonyms: marginal placenta or low placental insertion • Key Features 2-4: Receive 1/3 point for listing each of the following: 1. Avoid performing a pelvic exam, 2. Avoid discharging from clinic, 3. Order coagulation tests and cross match • Total Score for Problem: Add scores for items 1 and 2 and divide by 2 (range: 0 - 1)

19. Steps to Develop a Clinical Problem Based Exam • Define the domain of clinical problems to be sampled by the exam • Develop an exam blueprint to guide selection of clinical problems • Develop a key-feature problem for each clinical problem selected • define clinical situation for the problem (e.g. single typical problem, life-threatening situation etc.) • define key features of the problem • select a clinical case to represent the problem and write scenario • write exam items for case; in general one item for each key feature • select suitable format for each item (e.g., write-in or mcq) • develop scoring key for each item • pilot test items for item analysis data to guide refinement

20. Interim Summary of Session • Session thus far • Two uses of knowledge and performance assessments: Formative and Summative • Validity of all assessment techniques • Comparisons among 6 assessment techniques • Take and critique a 14 item multiple choice exam • Write a Key Features item • Coming up • View and critique a videotaped student-patient interaction as part of an OSCE

21. Schematic Diagram of a 9 Station OSCE Start 1 2 3 4 5 9 End 8 7 6

22. Widening the Lens on SP Assessment1 • Traditional scoring of SP assessment focuses on numerical data typically from checklists • Dimensions of the SP exam • basic science knowledge (organize the information) • physical exam skills (memory of routines) • establishing a human connection • role of the student (appear knowledgeable) • existential dimension of the human encounter (balance one’s own beliefs with the patient’s) • Clinical competence – mixture of knowledge and feeling, information processing and intuition 1. Rose, M. & Wilkerson, L. Widening the Lens on Standardized Patient Assessment: What the Encounter Can Reveal about the Development of Clinical Competence, Acad. Med. 76(8), 2001.

23. Interim Summary of Session • Session thus far • Two uses of knowledge and performance assessments: Formative and Summative • Validity of all assessment techniques • Comparisons among 6 assessment techniques • Take and critique a 14 item multiple choice exam • Write a Key Features item • View and critique a videotaped student-patient interaction • Presentation: Widening the lens on SP assessment • Coming up • Improving clinical performance assessment systems

24. Dr. Tough’s Memo regarding Dr. Will E. Makit (PGY 2) “The performance of Dr. Makit in General Surgery this month has been completely unsatisfactory. Every member of the clinical faculty who has had any contact with him tells me of his gross incompetence and irresponsibility in clinical situations. This person is an embarrassment to our school and our training program. I spoke to him about his performance after I talked with you several weeks ago and he told me that he would improve it. There was no evidence that he made any effort to improve. There is no way this can be considered a completed or satisfactory rotation in General Surgery. In fact, he is the most unsatisfactory resident who has rotated through our service in the last five years, and his behavior is an appalling example to the rest of our housestaff.” ************************************************************* Your 1. Refer the problem to the Resident Education Action? Committee for an administrative decision. 2. Assign Dr. Makit to a rotation with Dr. Insightful as the Attending Faculty.

25. Dr. Insightful’s Phone Comments regarding Dr. Makit

26. Resident Performance Assessment System Organizational Infrastructure

27. A. Department’s Organizational Infrastructure • Department head emphasizes completing and returning PA forms • Consequences for evaluators who don’t complete PA forms • PA form is brief ( < 10 competencies) • Don’t request pass/fail judgment by individual faculty • Evaluators trained to use PA form & criteria • Evaluators believe they will be supported when writing honest appraisals • Specific staff assigned to monitor compliance in returning forms • Program Director alerted immediately when a returned form reflects cautionary info Organizational Infrastructure

28. Evaluator Role in a Performance Assessment System

29. B. Evaluator Role: Communicate Expectations and Observe Performance • Communicate Expectations • Consensus among evaluators about service and education expectations • Residents are crystal clear about service and education expectations • Observe Performance • Evaluators observe resident multiple times before completing PA form • Appraise only performance directly observed • Other staff (e.g., nurses) complete PA forms Communicate Expectations Observe Performance

30. B.Evaluator’s Role: Interpret and Judge Performance • Evaluators agree on what behaviors constitute outstanding, ‘average’, and marginal performance • When facing a marginal resident, evaluators record rationale for their judgment and info surrounding the event • Evaluators record their interpretation of the performance soon after behavior occurs diagnose performance (quality wnl ?)

31. B.Evaluator’s Role: Coach Resident • Evaluators aware of difference between corrective feedback, criticism and compliments • Faculty actively coach residents in timely manner • Residents are encouraged to ask for feedback • Evaluators regularly invite self-assessment from residents before giving feedback Coach Resident

32. B. Evaluator’s Role: Communicate Performance Information and Complete PA Form • Communicate Performance Info • Communicate incidents that are significantly negative or positive • Document in writing even single instances of poor or inappropriate behavior • Complete PA Form • Evaluators write specific narrative comments on PA forms • Evaluators forward completed PA forms to Director in timely way Communicate performance information (to whom ?) Complete PA Form

33. Program Director Role in a Clinical Performance Assessment System

34. C. Program Director’s Role: Monitor and Interpret Appraisals • Recognize evaluator rating patterns (stringent vs. lenient) to accurately interpret PA • Contact evaluators to elicit narrative info when absent to substantiate a marginal PA • Store PA forms in residents’ files in a timely manner • Summarize PA data to facilitate decision making by Resident Education Committee • Keep longitudinal records of PA data to develop norms for the PA form

35. C. Program Director’s Role: Committee Decision • PA decisions are a collaborative process involving multiple faculty • Seven or more PA forms per resident are available when admin decisions are made • Sufficient written narrative documentation is available when admin decisions are made committee decision

36. C. Program Director’s Role:Formally Inform Resident • Residents are given a summary of their performance every six months • Evaluators have written guidelines outlining what must legally be in a probation letter • Evaluators know what documentation is needed to ensure adequate due process • Each resident receives an end of program performance evaluation inform resident

37. Formative Evaluation: Diagnostic Checklist for Resident Performance Assessment System

38. Research: Improving Resident Performance Appraisals 1 • Organizational Infrastructure • Discussed PA problems at department meetings • Appointed a task force to review PA problems and propose solutions • Revised old appraisal form • Pilot-tested and adapted the new appraisal form • Evaluator Role • Provided evaluators with examples of behaviorally-specific comments • Results • Increased # of forms returned, # forms with behaviorally-specific comments, and # of administrative actions by prog. 1. Littlefield, J. and Terrell, C. Improving the quality of resident performance appraisals, Academic Medicine, 1997: 72(10) Supplement, S45-47.

39. Research: Improving Written Comments byFaculty Attendings1 • Organizational Infrastructure • Conducted a 20 minute educational sessions on evaluation and feedback • 3 by 5 reminder card and diary • Results • Increased written comments specific to defined dimensions of competence • Residents rated quantity of feedback higher and were more likely to make changes in clinical management of patients 1. Holmboe, E., et.al. Effectiveness of a focused educational intervention on resident evaluations from faculty. J. Gen Intern Med. 2001: 16:427-34.

40. Research: Summative Evaluation of a PA System1,2 • 1. Littlefield, J.H., Paukert, J., Schoolfield, J. Quality assurance data for resident global performance ratings, Academic Med., 76(10), supp., S102-04, 2001. • 2. Paukert, J. et. al., Improving quality assurance data for resident subjective performance assessment, manuscript in preparation

41. Recall a medical student or resident whose performance made you uneasy. • What behavior or event made you uneasy? • What action did you take? a. Talk with faculty colleagues about your concerns b. Write a candid performance appraisal and send it to the course/residency director 3. If you wrote a candid appraisal, did an administrative actionoccur related to the student/ resident?

42. Goals: Assessment of Knowledge & Performance • Clarify two distinct uses for assessments of clinical knowledge and performance • Define two aspects of validity for all assessment methods • Compare and contrast 6 techniques for assessing clinical knowledge and performance • Identify poorly written multiple choice test items • Write a key features test item • Describe a role for narrative comments in scoring interactions with Standardized Patients • Describe three elements of a clinical performance assessment system • Critique a clinical performance assessment system that you use