Assessment in the Workplace

1. John Norcini, Ph.D. Assessment in the Workplace

2. Workplace-Based Assessment • Overview • Why workplace assessment? • Description of some instruments • Faculty preparation • Feedback to trainees • Implementation

3. Clinical Skills are Important • Interview and PE are the primary source of diagnostic information • Compared to lab studies and imaging • Data gathering mistakes are the most common cause of diagnostic errors • Good communication skills • Improve health status of patients • Increase the efficiency of care • Decrease the likelihood of malpractice suits

4. Clinical Skills are Deficient • Studies document clinical skills deficiencies • Ascultatory skills in trainees (Mangione, 1997) • History-taking/preventive health screening among primary care doctors (Ramsey, 1999) • Missed physical findings among residents (Reilly, 2003) • Errors during procedures (Tang et al., 2005)

5. Formative Assessment and Feedback are Lacking • There is a lack of formative assessment and feedback in workplace • Medical students • Structured observation done for only 7-23% of students (Kassebaum & Eaglen, 1999) • Only 28% of IM clerkships include formative assessment strategy (Kogan & Hauer, 2006) • Postgraduate trainees • 82% were observed only once (Day et al., 1990) • 80% observed never or infrequently (Isaacson et al., 1995)

6. Formative Assessment and Feedback are Lacking • Critical to learning and have a significant influence on achievement • General education (Hattie, 1999) • Meta-analysis of 12 meta-analyses • Feedback is among the largest influences on achievement (ES=.79) • Medical education (Veloski et al., 2006) • Feedback alone effective is effective in 71% of studies

7. Massed Training Spaced Training Sessions Few, Intense Many, Spread out Speed Confidence Satisfaction Retention Performance Formative Assessment and Feedback are Lacking

8. Massed Training Spaced Training Sessions Few, Intense Many, Spread out Speed Faster Confidence Higher Satisfaction Greater Retention Longer Performance Better Formative Assessment and Feedback are Lacking

9. Formative Assessment and Feedback are Lacking • Retrieval of information or a performance enhances learning • Students read a passage (Roediger & Karpicke, Psych Science, 2006) • Group 1 took three tests on the passage • Group 2 re-read the passage carefully three times • On a test one week later, Group 1 did better • Students read science text (Karpicke & Blunt, Science, 2011)

10. Traditional Workplace Assessment is Flawed • Traditional CEX • One examiner observes a trainee interact with an unfamiliar (in)patient • Trainee does a complete Hx/PE, presents findings, management plan, written record • Examiner rates along several dimensions • Takes about two hours • 82% of trainees undergo a CEX in their first year

11. “One third of the mice used in the experiment were cured by the test drug; One third of the test population were unaffected by the drug and remained in a moribund condition; The third mouse got away.” Erwin Neter Traditional Workplace Assessment is Flawed • The trainee is evaluated with only one patient • Physician performance varies considerably from patient to patient

12. Traditional Workplace Assessment is Flawed The trainee is evaluated by only one examiner Examiners differ in stringency “You get 15 Democrats in a room and you get 20 opinions.” Senator Patrick Leahy

13. Traditional Workplace Assessment is Flawed Most real physician-patient encounters are short and focused The task is artificial "Reality is merely an illusion, albeit a very persistent one." Albert Einstein

14. Foundation Programme is best package of methods Peer assessment CbD DOPs Mini-CEX Workplace-Based AssessmentMethods • Focus on formative assessment of clinical skills • Respond to the assessment problems of the traditional CEX • Respond to the educational problems of the workplace • Requires observation and feedback

15. Description of Peer Assessment • Process • Trainee nominates 8 assessors and self-rates • Web-based (now) • Assesses clinical and generic skills • Collated centrally • Trainee given self-ratings, assessor ratings, national mean ratings, and comments • 2 assessments per year

16. Description of CbD • Chart Stimulated Recall • Process • Trainee picks 2 case records • Assessor selects one • Discussion centered on the trainee’s notes • Assessor rates Diag, Treat, Planning, Prof, etc. • Takes 15-20 minutes • 6 assessments/year

17. Description of DOPs • Process • Assessor observes a trainee with a patient • Trainee performs a procedure • Assessor rates Prep, Sedation, Asepsis, Technical Skill, etc. and provides feedback • Takes 15-20 minutes • 6 assessments/year

18. Description of mCEX • Process • Assessor observes a trainee with a patient • Trainee performs a focused clinical task • Assessor rates Hx, PE, Comm, CJ, Prof, Org/Eff and provides feedback • Takes 15-20 minutes • 6 assessments/year

19. Challenges Not many trainees will be considered unsatisfactory There remains a need for summative assessment "Everywhere I go I'm asked if I think the university stifles writers. My opinion is that they don't stifle enough of them." Flannery O'Connor

20. Challenges Trainees have some control over who examines them and indirectly over the content of the assessment The assessment might be biased in their favor “It is hard to believe that a man is telling the truth when you know that you would lie if you were in his place.” H. L. Mencken

21. Challenges Standards across programmes will not be equivalent Results will not be useful for national ranking of trainees “Equal opportunity means everyone will have a fair chance at becoming incompetent.” Laurence J Peter

22. Challenges • Faculty are not trained to use the methods • Faculty development is needed • Feedback is often limited and not as helpful as it could be • Characteristics of good feedback • Implementation is sometimes difficult • Process for encouraging implementation

23. Video Exercise Pt presents with chest discomfort that started in the morning. ER setting and trainee needs to decide if pt is to be admitted. Rate on a scale of 1-3 Unsatisfactory 4-6 Satisfactory 7-9 Superior Not able to assess Rate the following Interviewing PE Professionalism Clinical judgment Counseling Org/efficiency Overall competence Video2

24. Average Rating of ‘4’ • Primary • Did not ask if pt had chest pain now • Did not ask about duration of pain • Did not ask what made pain better • Did not ask about prior episodes • Secondary • Did not ask father’s age at MI • Did not ask about occupation • Lacked pt centeredness • Did not ask about heart disease in other family • Did not ask if pt had questions

25. Video Exercise • New pt with 2-day Hx of productive cough and mild shortness of breath. Has felt warm at home, smokes a pack/day and has an 80 pack-year Hx. BP=110/72mm Hg and temp=101.5F. Rate on a scale of • 1-3 Unsatisfactory • 4-6 Satisfactory • 7-9 Superior • Not able to assess • Rate the following skills • Interviewing • PE • Professionalism • Clinical judgment • Counseling • Org./Efficiency • Overall competence Video 6

26. Average Rating of ‘7’ • Primary • Did not check respiratory rate • Did not examine nasal passages • Pulmonary: Did not listen anteriorly with stethoscope • Cardiac: Did not assess PMI • Secondary • Did not listen to complete respiratory cycle before moving on • No thyroid exam • Posterior lymph node exam not complete • No abdominal exam

27. Video Exercise • Can you identify any problems with the way the faculty conducted the exercise? (Video 10) • Late • Poor explanation to patient of his role • Poor positioning to observe the physical exam • Disrupts the blood pressure measurement by washing his hands • Disrupts the eye exam by moving around and inserting himself • Disrupts the eye exam by asking the patient questions • Distracted by the knock on the door

28. Faculty Preparation • Some characteristics of effective programs (Skeff et al, 1997) • Addresses the needs of the faculty participants • Systematic approach • Takes the workplace into account • Emphasizes theory and practice • Opportunity for practice and feedback • Builds relationships among participants

29. Faculty Preparation • Direct Observation of Competence training (Holmboe, Hawkins, Huot, 2004) • Behavioral observation • Know what to look for • Prepare resident and patient • Minimize intrusiveness and interference • Performance dimension training • Decide which dimensions of performance are important

30. Faculty Preparation • Direct Observation of Competence training • Frame of reference training • Improve accuracy and discrimination • Reduce stringency differences • Practice • Workshop • Didactic mini-lectures • Small group and videotape evaluation exercises • Practice with standardized residents and patients

31. Faculty Preparation • Study of the DOC model (RCT) • Faculty who underwent training • Thought the workshop was excellent • Felt more comfortable performing direct observation • Were more stringent than control group faculty

32. Feedback Trainees are rarely observed in patient encounters Limits the opportunity for evaluation and feedback When observed feedback is sometimes poor Mini-CEX requires observation and offers the possibility for educational feedback “The belief that all genuine education comes about through experience does not mean that all experiences are genuinely or equally educative.” John Dewey

33. Video Exercise Pt has hyperlipidemia and failed at weight loss and dietary restriction. Cholesterol is 285 (45, 170, 206). Risk factors: age (>45), hypertension, family Hx, and smoker. She needs to start statins. Rate as 1-3 Unsatisfactory 4-6 Satisfactory 7-9 Superior Rate the following skills Interviewing PE Professionalism Clinical judgment Counseling Org./Efficiency Overall competence What feedback would you give the trainee?

34. Feedback Feedback needs to address three learner questions (Hattie & Timperley, 2007) Where am I going? How am I going? Where to next? Factors affecting the impact of feedback Focus Trainee Technique Creating an action plan Mentoring

35. Feedback: Focus • Four areas of focus (Hattie & Timperley, 2007) • Feedback about the task • Quality of the performance • Feedback about the process of the task • Encourages deeper learning and transfer • Feedback about self-regulation • Help-seeking, self-assessment, self-efficacy, etc. • Feedback about the self as a person • Rarely effective

36. Feedback: Trainee • Response to feedback is influenced by • Trainee’s level of achievement (Shute, 2008) • Culture (DeLuque & Sommer, 2000) • Collectivist vs. individualist • Perceptions of accuracy (Sargent et al, 2005) • Perceptions of credibility and trustworthiness (Albright et al, 1995) • Perceptions of usefulness (Brett et al, 2001)

37. Feedback: Technique Technique influences impact (Hewson et al, 1998) Establish an appropriate interpersonal climate Use an appropriate location Elicit the learner's thoughts and feelings Reflect on observed behaviors Be nonjudgmental Be specific Offer the right amount of feedback Offer suggestions for improvement

38. Feedback: Action Plan and Mentoring Creation of an action plan leads to change Answers the “Where to next?” question Feedback alone does not cause change, it is the goals that people set in response to feedback (Locke et al, 1990) Mentoring increases the likelihood of change following feedback Broad management literature (Luthans et al., 2003; Walker et al, 1999)

39. Feedback: Summary Plan for good feedback (Holmboe et al., 2004) Provide an assessment of strengths and weaknesses Enable learner reaction Encourage self-assessment Develop an action plan

40. Small Group Discussion:Implementation • Some groups have difficulty getting clinical faculty to be involved with workplace-based assessment • What do you do regarding this problem? • What are some of the obstacles faculty face? • Do you know of any faculty that have overcome the obstacles? • How have they overcome these obstacles?

41. Implementation Issues • What are your concerns about implementation? • I don't have enough time to do this • There is too much paperwork • What is the evidence? • What is the purpose? • What is the focus? • What might address these concerns?

42. Implementation Strategies • Make it mandatory • Provide the faculty • Time/compensation • Training/practice • Documentation • Clear guidelines • Feedback • Recognition • Use core faculty/champions • Increase efficiency • Adapt locally • Electronic

43. Positive Deviance “In every community or organization there are certain individuals or groups whose uncommon practices/behaviors enable them to find better solutions to problems than their neighbors or colleagues who have access to the same resources” Work of Jerry and Monique Sternin www.positivedeviance.org

44. Positive Deviance:Vietnam Case Study • 63% of children in 4 villages were malnourished • Community volunteers charted nutritional status and surveyed common practices • Identified-observed children from very poor families who were well nourished • Collected tiny shrimps, etc. from the rice paddies • Used greens from sweet potato tops • Engaged in good hygiene • Fed children 4-5x a day instead of 2x • Volunteers taught caretakers how to prepare/feed meals • Created group meals where caretakers were required to bring shrimp, etc. as the price of admissions • Routinized trips to the rice paddy

45. Positive Deviance:When to Use It • The problem is not completely technical and it requires social-behavioral change • Other solutions have not worked • Positive deviants exist • There is leadership commitment to change

46. Positive Deviance:Principles • Emphasis on behavior • Community • Discovers existing, uncommon, successful behaviors • Creates a plan for implementation • Recognizes that they can get better results • Owns the process • Creates its own performance indicators

47. Positive Deviance: Process • Develop a team • Define the problem • Identify deviants • Identify uncommon approaches • Community designs an action plan • Evaluate the effectiveness of the plan • Disseminate the results

48. Positive Deviance:Summary • Implementation of workplace-based assessments can be challenging • Positive deviance offers an approach • Define the problem • Determine if there is anyone who exhibits the behavior • Discover their uncommon behaviors/strategies • Develop and implement local solutions

49. Summary Why workplace assessment? Provides the formative assessment and that is critical to learning Faculty preparation is critical for both feedback and assessment Positive deviance is one model for acquiring faculty cooperation