Developing High Quality Clinical Skills Assessments

1. Developing High Quality Clinical Skills Assessments University of North Carolina – Chapel Hill School of Medicine November 10, 2011 Ann Jobe, MD, MSN Clinical Skills Evaluation Collaboration Philadelphia, Pennsylvania

2. The most important consideration in developing a clinical skills assessment is to keep in focus the purpose of the assessment

3. Formative or Summative? Formative: to provide feedback for improvement of performance, usually midway through a course of study. Summative: to distinguish between those who are competent, and those who aren’t, usually at the end of a course of study.

4. Test Design • Begin at the beginning… • What are you trying to assess? • What is the level of the examinee? • How much time will the examinee have? Can they complete the task in the time allotted?

5. Test Design • What are you trying to teach and assess? • Objectives for assessment – history taking skills, PE skills, communication skills, data interpretation skills? • What is the level of the examinee? • Focus and length of checklists will be different for 1st year medical students and individuals seeking board certification

6. USMLE Step 2 Clinical Skills Purpose is to assess ability to: • Gather information from patients • Correctly perform physical examination maneuvers • Synthesize and communicate findings to patients and colleagues Examinees are moving to supervised patient care (PGY-1)

7. Development Of An Examination Blueprint • A test blueprint defines the requirements for each examination, regardless of where or when it is administered. • Commonly seen cases.

8. Blueprint (continued) • Identify the criteria used to define an exam blueprint • Examination length • Case content • Examinee tasks (e.g. history, physical, communication) • Setting: inpatient or outpatient • Patient gender, age

9. Case Selection • Documentation of commonly seen cases • if specialty focused assessment, cases will be those common to that specialty • Most common presenting complaints in clinical setting • Health Department population statistics – especially for region, state or local area

10. Case Pool Size • Dependent on blueprint criteria • Security concerns (more is better!)

11. Case Content Cardiovascular Respiratory Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Other Case Acuity Acute Subacute/Chronic Form Patient age Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 + Patient Gender Male Female

12. Blueprint Criteria Define Case Needs • This information is provided to faculty to guide their case development • Case content: Gastrointestinal • Case acuity: Acute • Patient age: 18-44 • Patient gender: Female • Complete a “Medically Relevant Case Details” sheet

13. Case Development Committee • Physicians (medical school faculty): content experts • SP trainers: training experts • SPs: portrayal experts • Process evolved from experience

14. Usually break into several groups Each group focuses on developing several cases to fit blueprint needs

15. Case Development Process • Chief Complaint • “Medically Relevant” case details • Patient Personality Profile • Case checklist • Examinee instructions • Patient note development

16. Chief Complaint/ Differential Diagnoses • Brainstorming process • List all of the possibilities • Then narrow the choices as development proceeds

17. Medically Relevant Case Details • Patient /Case Name • Race • Any • African American • Caucasian • Other _________________ • BMI • Any • Weight proportionate to height • Overweight: > 24 • Other

18. Medically Relevant Case Details • Gender • Male Female Either • Age Range • 10 year age range – example 30 years old = 25-35 • Acuity • Acute • Subacute/Chronic • Category (Blueprint) • Exclusionary Medical conditions/scars • Differential Diagnosis

19. Medically Relevant Case Details • Communication Tasks • Fostering the Relationship • Gathering Information • Providing Information • Making Decisions • Supporting Emotions • Overview of Case Stimuli for Communication Tasks • Thread of Encounter • Primary MD

20. Personality Profile Giving life and individuality to the patients who are being seen – more like “real patients” Not verbatim statements from a script – more general ideas of each patient’s perspective

21. Personality Profile Who am I and how would I describe myself? Why am I here? What made me come in today? What do I think is going on and why? What are my expectations for the visit and how will I react if my expectations are not met?

22. Personality Profile What questions do I have? What concerns/anxiety/fears do I have? How has this illness impacted my life? What do I bring with me? What do I look like? How do I behave/what’s my communication style (mood/attitude related to my illness)?

23. Personality Profile What is my level of trust of the medical profession? What type of communication style do I appreciate in a health care provider? How will I respond to different styles of communication? How much information do I want the doctor to provide?

24. Personality Profile What words won’t I understand (jargon)? What are my own words I use to describe what is going on? How motivated am I to accept medical advice? How do I feel about the use of complementary/alternative medicine?

25. Case Checklist • Recording instrument • Checklist length • One concept per item • Evidence-based items • Use lay language Example: Onset (“I’ve been coughing for about a month.”)

26. Refining the Checklist • Brainstorm a large number of items • Role play • Checklist revised • A – essential • B – important • C – relevant • F - delete

27. Examinee Instructions • Name • Age • Setting • Presentation problem • Vital signs • Examinee Tasks • Refers to the skills that the examination assesses

28. Role Play with SP • A physician from another group “sees” the patient • Following the encounter, the physician provides his/her differential diagnoses (see if it matches what the group selected): face validity • Revise the checklist based on observing the physician

29. Role Play with SP • This allows an SP to provide feedback on any difficulties that arise in portrayal of the case – especially if an SP needs to repeat the case several times in an assessment session

30. Development of Patient Note • Key essentials and case-based scoring guidelines

31. Questions?