Competencies and Milestones: The Next Step in Meaningful Assessment of Learners

1. Competencies and Milestones: The Next Step in Meaningful Assessment of Learners • Robert Englander, MD MPH • Duke University School of Medicine • April 8th, 2013

2. Objectives for the afternoon • Advance your working knowledge of CBME • Understand the context for the development of specialty-specific milestones • Compare the current state of assessment to the milestones • Experience team-based learning • Have fun!

3. Process for TBL • Form teams for a longitudinal course/activity • Pre-reading for each session • Begin the session with a closed-book test • Discuss the test as a team arriving at a consensus answer for each team • Poll the teams for the correct answers (different methods) • Review subject matter for any question with <80% teams answering correctly • Apply knowledge with case-based team exercise

4. Central Tenet of CBME

5. The Vision • Physicians will spend their careers, from premed to exit from practice, on a developmental trajectory building mastery in 8 domains of competence

6. The “Complete Physician” *Outcome of IPEC **Outcome of Pediatrics Milestone Project

7. The Paradigm Shift

8. Framework for Evaluation of Competence

9. Framework for Evaluation of Competence (cont’d)

10. Competenciesfor the Domain of Patient Care-1 • Perform procedures of the specialty • Organize and prioritize responsibilities* • Gather essential information • Interpret lab data, imaging studies and other tests • Make informed diagnostic/therapeutic decisions • Develop and carry out management plans *Black font color denotes one of the 21 competencies reported to the RRC for Pediatrics

11. Competenciesfor the Domain of Patient Care-2 • Counsel and educate patients and families • Provide referral that insures seamless transitions of care • Provide health maintenance/anticipatory guidance • Provide role modeling • Perform supervisory responsibilities commensurate with role

12. Competencies for the Domain of Medical Knowledge • Demonstrate an investigational and analytic approach to clinical situations • Apply the established and emerging biophysical, clinical, epidemiological, and social-behavioral sciences appropriate to one’s discipline • Contribute to new health care knowledge and practices

13. Competencies for the Domain of Interpersonal and Communication Skills • Communicate effectively with patients, families, and the public • Communicate effectively with other colleagues within one’s profession, other health care professionals, and health related agencies • Work effectively with others as a member or leader of a health care team or other professional group • Provide consultation to other health professionals • Maintain comprehensive, legible, and timely records • Demonstrate sensitivity, honesty, and compassion in difficult conversations • Demonstrate insight into emotions to allow management of interpersonal interactions

14. Competencies for the Domain of Professionalism • Demonstrate compassion, integrity and respect for others • Demonstrate responsiveness to patient needs that supersedes self-interest • Demonstrate respect for patient privacy and autonomy • Demonstrate accountability to patients, society, and the profession • Demonstrate sensitivity and responsiveness to a diverse patient population • Demonstrate a commitment to ethical principles (both patient care and business practices)

15. Competencies for the Domain of Systems-based Practice • Work effectively in various health care delivery settings • Coordinate patient care within the health care system • Incorporate considerations of cost awareness and risk-benefit analysis in patient and/or population-based care • Advocate for quality patient care and optimal patient care systems • Participate in identifying systems errors and implementing potential systems solutions • Perform administrative and practice management responsibilities commensurate with one’s role

16. Competencies for the Domain of Practice-based Learning and Improvement • Identify strengths, deficiencies and limits in one’s knowledge and expertise • Set personal learning and improvement goals • Identify and perform learning activities that match one’s needs • Systematically analyze one’s practice and make improvements • Incorporate formative feedback into practice

17. Competencies for the Domain of Practice-based Learning and Improvement (2) • Locate, appraise, and assimilate evidence from scientific studies related to a patient’s and/or population’s health • Use information technology to optimize learning • Participate in the education of patients, families, students, trainees, and other health professionals • Obtain and utilize information about individual patients, populations, or communities to improve care • Continually identify, analyze, and implement new knowledge, standards, products, or services that may impact a patient’s or population’s health

18. Competencies for the Domain of Interprofessional Collaboration • Work with individuals of other professions to maintain a climate of mutual respect and shared values • Use knowledge of one’s own and others’ roles to assess and address health care needs of individuals and populations • Communicate with other health professionals to optimize health maintenance and treatment of disease in patinets/populations • Perform effectively in different team roles to plan/deliver patient/population-centered care that meets the IOM quality aims *Interprofessional Education Collaborative Expert Panel (2011). Core Competencies for Interprofessional Collaborative Practice: Report of an Expert Panel. Washington, DC: Interprofessional Education Collaborative.

19. Competencies for the Domain of Personal and Professional Development • Know one’s limits and engage in help-seeking behaviors • Demonstrate a healthy response to stress • Manage conflict between personal and professional responsibilities • Practice flexibility and maturity in response to change • Demonstrate trustworthiness • Demonstrate leadership that ultimately improves patient care • Demonstrate confidence • Manage Uncertainty

21. How We Assess Learners is Key • Sharing perspectives to get us to the same mental image of learner behaviors • Sharpening focus to optimize observation • Honing our observation skills

22. Observation Skills Video

23. Current state of assessment tools versus the milestones • A head-to-head match-up

24. Global Rating: Patient Care

26. Rate MS 3 Performance Using a 9-point Likert Scale • Unsatisfactory • Unsatisfactory • Unsatisfactory • Marginal • Satisfactory • Satisfactory • Superior • Superior • Superior

27. Rate PGY-2 Performance Using a 9-point Likert Scale • Unsatisfactory • Unsatisfactory • Unsatisfactory • Marginal • Satisfactory • Satisfactory • Superior • Superior • Superior

28. “Intermittent interaction based on satisfaction of rules” “continuous dialogue based on a desire to improve educational outcomes” We must agree on the “Milestones” of competency development in each discipline. We must agree on and implement common evaluation tools in each discipline to…document resident achievement of these milestones ACGME Bulletin 9-2008

29. The Milestones Project Charge • Refine the competencies in the context of the specialty • Set Performance Standards • Identify or develop tools for assessment of performance

30. Pediatricians LOVE Milestones !

31. Guiding Principles • The 6 domains of competence are necessary, but notsufficient • Milestones must be grounded in the literature • Milestones describe sequential behaviors, providing a learning roadmap for trainees • Milestones span the continuum from UGME to CME/MOC

32. Pediatrics Milestones: Process Harris, I.B., Deliberative inquiry: the art of planning, in Forms of Curriculum Inquiry, E.C. Short, Editor. 1991, State University of New York: Albany, NY. p. 285-307. “Succession of lenses” Comb the literature Build upon relevant models and theories Revise to accommodate “lenses”

33. Pediatrics Milestones: Process • Sought the ontogeny of observable behaviors for each sub-competency

34. The Product A series of milestones for each of the 51 competencies

35. Pediatrics Milestones Construction of a Pediatrics Milestone • Background/synopsis of literature • References • Milestones • A sequence of narrative descriptions of observable behaviors at advancing levels of development across the continuum of education, training and practice

36. Knowledge Application Exercise #1: Example Competency in the Domain of Patient Care Making informed diagnostic and therapeutic decisions that result in optimal judgment

37. 1 “First level” • Recalls and presents clinical facts in the history and physical in the order they were elicited without filtering, reorganization or synthesis • Non-prioritized list of all diagnostic considerations rather than the development of working diagnostic considerations • Difficulty developing a therapeutic plan • Summary: Regurgitates history and physical and then looks to supervisor for synthesis and plan.

38. 2 “Second Level” • Focuses on features of the clinical presentation, making pattern recognition elusive and leading to a continual search for new diagnostic possibilities. • Often reorganizes clinical facts in the history and physical exam to help decide on clarifying tests to order rather than to develop and prioritize a differential. • This often results in a myriad of tests and therapies and unclear management plans since there is no unifying diagnosis • Summary: Jumps from information gathering to broad evaluation without focused differential

39. 3 “Third Level” • Abstracts and reorganizes elicited clinical findings in memory, using semantic qualifiers to compare and contrast the diagnoses being considered when presenting or discussing the case. • Well synthesized and organized assessment of the focused differential diagnosis and management plan • Summary: Synthesizes information to allow a working diagnosis and differential diagnosis that informs the evaluation and management plan • .

40. 4 “Fourth Level” • Reorganized and stored clinical information leads to early directed diagnostic hypothesistraining with subsequent history, physical, and tests used to confirm this initial schema • Able to identify discriminating features between similar patients and avoid premature closure • Therapies are focused and based on a unifying diagnosis, resulting in an effective and efficient diagnostic work-up and plan • Summary: Rapid focus on correct working and differential diagnosis allows efficient and accurate evaluation and management plan

41. Which Milestone Best Reflects MS 3 Performance Level? PGY-2? • Milestone One • Milestone Two • Milestone Three • Milestone Four

42. Knowledge Application Exercise #2: Professionalism Milestones • Assign a reporter and a recorder • View the professionalism video • Review the assigned professionalism milestones • Individually decide which level of performance best reflects the learner • Discuss as a team, including feedback you would provide the learner to get her to the next level of performance • Report out the consensus level and one piece of feedback.

43. Advantages of General Competencies/Milestones • Insure comprehensive conversation • Identify important physician attributes as such • Improvement over “the mist of holistic waffle about professional experience and the ineffability of…intuitive wisdom.”1 • Focus assessment on achievement of consensus competencies. 1) Cooke M, Irby DM, et al. (2010). Educating Physicians: A Call for Reform of Medical School and Residency. San Francisco, Jossey-Bass.

44. Reductionist vs Holistic Paradigms

45. Resolution • Provide an integrative construct that places the competencies and milestones in the context of clinical care

46. Putting it Back Together….

47. Questions?

48. THANKS!