Student Assessment: New Ideas and Old Basics

1. 26 September 2007 Student Assessment:New Ideas and Old Basics Louis Pangaro, MD Professor and Vice-chair for Educational Programs Department of Internal Medicine Uniformed Service University of the Health Sciences

2. “clinical assessment” • By teachers (house staff and faculty) • On clinical rotations • Based on words = descriptors • Using words = descriptive • In vivo (vs. in vitro end of course/year)

3. What’s old? • Suspicion of grades by teachers • Teachers reluctance to be direct (honest?) • Belief that grading by teachers is subjective • Lawsuits about low grades • Jamieson, Guidebook for Clerkship Directors, 2005.

4. What’s basic? • We have an obligation • Fairness • Mentoring • The Hippocratic Oath • Professionalism • Duty and expertise (Pellegrino)

5. How it looked in 1987 • lack of meaningful comments by evaluators • insufficient definition of evaluation criteria • too much inter-observer variability • late submission of evaluations • delay in feedback to students Tonesk X, Buchanan RG. J Med Ed. 1987

6. How it looked in 2001 • “Areas of weakness in current [clinical] evaluation models include psychometric properties associated with the tools, namely their questionable reliability and validity.” • Turnbull, International Handbook of Research in Medical Education, 2002.

7. How it looks in 2007 • “Constructive criticism is hard to come by..” • “…Candor is at least as painful to the provider as to the recipient…” • “…faculty members feel uncomfortable inflicting pain even in a good cause like student improvement.” • HMS Student Handbook, 2006-2007

8. focus • The emotional issues for teachers and learners in the grading process. • Dealing with barriers to candor.

9. Quality: minimizing unwarranted variation in physician performance. • Wennberg • Faculty Development: minimizing unwarranted variation in teacher performance.

10. What’s new • A believe that quality methods can apply to teaching. • Search for more rigor. • “Best Evididence Med Ed” (BEBM) • “Med Ed Research Certificate” • MPH, MHPE programs

11. Today: describing success in clinical evaluations

12. New focus on evaluation in clincial setting: • Attention to professional traits (Papadakis) • Lawsuits by patients

13. Context of new methods • Objective Structured Clinical Exams • 3600 assessments • Portfolios • Descriptive vocabularies • To get more inter-rater agreement • To calibrate differences between levels of performance

14. FOCUSED PHYSICAL EXAMINATION • [Norm] • Exam is generally appropriate in scope and technique. Identifies major abnormalities and pertinent normal findings, only occasionally missing elements. Exam linked to history. Appropriate for level of training. • HMS Clerkship Grading Form

15. FOCUSED PHYSICAL EXAMINATION (2) • [Poor] • Consistently misses important findings and often does not make appropriate connection between history and physical. Often uses faulty or inappropriate technique. Not organized or thorough.

16. FOCUSED PHYSICAL EXAMINATION (3) • [Excellent] • Exam is consistently superior. Uncovers subtle and important findings, incorporating advanced techniques where appropriate. Exceptionally organized and thorough, even on difficult cases.

17. American Board of Internal Med 1 2 3 4 5 6 7 8 9 Average and 2 SDs Minimal acceptable

18. ACGME Outcomes Project • New emphasis on outcomes (vs curriculum) • “long-term effort designed to emphasize educational outcome assessment in residency programs and in the accreditation process.”

19. ACGME: Outcomes: a cultural shift • Designing Curriculum is no longer enough • Results must be demonstrated. • A shift from process to product. • Content expertise not enough; need pedagogic expertise

20. ACGME “Competencies”(1999) • Medical Knowledge • Interpersonal & communication skills • Professionalism • Patient Care • System-based Practice • Practice-based learning & Improvement

21. the ACGME has spoken !

22. Medical Expert Communicator Collaborator Professional Manager Health Advocate Scholar CanMEDS (2005)

23. “R.I.M.E. Scheme” • Reporter • Interpreter • Manager/Educator Pangaro, Academic Med, 1999

24. Acceptance (1) Ob-Gyn “The RIME method is a valid, logistically feasible and acceptable way of assessing medical student clinical performance… …..minimizes disadvantages of descriptive evaluation, and maximizes the opportunity for accurate observations and helpful feedback.” APGO UME Taskforce: Espey et al, Am J Ob Gyn, 2007

25. Fairness Consistency Expectations

26. Principle 1 : Fairness • to society : • valid (not arbitrary) and • sensitive to detect marginal performers • to students : • know what’s expected, • timely feedback • to teachers : • know what observations to make • protected (legally, emotionally)

27. Principle 2 : Consistency • Reliability • Within teacher • Between teachers • In same rotation or across blocks • Between sites in same clerkship • Between disciplines • suitable for high-stakes decisions

28. strategy • Simplicity leads to acceptance and use • Acceptance to consistency • Consistency to fairness

29. Fairness & Consistency Reliability &Validity = Stability of measurement and strength of inferences from observations

30. “ Old” beliefs in our culture • Grading by teachers is subjective, Examinations are objective • Measurement > description, Numbers are > words

31. Framing the question more simply: • What do we expect of students? • Can we get all teachers to have the same expectations, and apply them consistently? • [A question of words and of conceptual frameworks]

32. Educational Goals Curriculum Evaluation Feedback / Grading community Depends on expectations

33. The goal: progressive independence of the learner Learner Teacher/program Content – Goals (Patients) after SFDP

34. Frames of referencefor expressing goals • Analytic • Developmental • Synthetic

35. goals for education (generic): • attitudes/behavior • skills • knowledge “KSA”

36. Bloom’s Taxonomies Cognitive Domain Psychomotor Domain Affective Domain

37. 1. Analytic expression of Goals • “ana-lytic”: takes the learner “apart” • into domains, categories • “attitude”, “skills”, “knowledge” • considered separately • generic terms • useful for discrete assessments

38. Curricular Goals: KNOWLEDGE The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following: • The capacity to recognize the limitations in one’s knowledge and clinical skills and to make a commitment to engage in lifelong learning • Knowledge of the normal structure and function of each of the major organ systems of the body and the current basic scientific mechanisms operative at the systemic, cellular, and molecular levels • Knowledge of the various causes (genetic, developmental, metabolic, toxic, microbiologic, immune, psychosocial, neoplastic, traumatic, and degenerative) of illnesses and diseases • Knowledge of the altered structure and function of the body and its major organ systems that are seen in various illnesses and diseases • Knowledge of the scientific method in establishing the causation of disease and efficacy of traditional and non-traditional therapies • Knowledge of health care policy and the economic, psychological, social, and cultural factors that affect health and health care delivery • Knowledge of the most frequent clinical, laboratory, radiographic, and pathologic manifestations of common as well as life threatening diseases • Knowledge about relieving pain and ameliorating the suffering of patients • Knowledge of the epidemiology of diseases and the systematic approaches useful in promoting health • Knowledge of techniques of patient education and counseling in basic lifestyle changes/prevention • Knowledge of and approaches to reduce the psychological and physical risks and stresses of the practice of medicine

39. Curricular Goals: SKILLS • The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following: • The ability to obtain an accurate medical history and the ability to perform both a complete and an organ specific examination, including a mental status examination (See appendix A) • The ability to perform routine technical procedures (See appendix B) • The ability to interpret the results and be aware of the indications, complications, and limitations of commonly used diagnostic procedures (See appendix C) • The ability to demonstrate knowledge of theories and principles that govern ethical decision making • The ability to reason deductively and inductively in solving clinical problems • The ability to construct appropriate differential diagnoses and treatment plans for patients with common conditions, both acute and chronic, including medical, psychiatric, and surgical conditions, and those requiring short- and long-term rehabilitation • The ability to recognize patients with immediate life threatening conditions regardless of etiology, and to institute appropriate initial therapy • The ability to recognize and outline an initial course of management for patients with serious conditions requiring critical care • The ability to communicate effectively, both orally and in writing, with patients, patients’ families, colleagues, and others with whom physicians must exchange information in carrying out their responsibilities • The ability to select appropriate tests for detecting patients at risk for specific diseases and to determine strategies for responding appropriately • The ability to retrieve, critically review, and effectively utilize biomedical information from electronic databases and other resources for solving problems and making decisions that are relevant to the care of individuals and populations • The ability to evaluate the economic, psychosocial, and cultural factors that impact the health of patients and families and to incorporate these into assessment and treatment plan

40. Curricular Goals: ATTITUDES • The School of Medicine will ensure that before graduation a student will have demonstrated, to the satisfaction of the faculty, the following: • A commitment to advocate the interests of one’s patients • Compassionate treatment of patients, and respect for their privacy and dignity • Honesty and integrity and dutifulness in all interactions with patients, their families, colleagues, and others with whom physicians interact • An understanding of, and respect for, the roles of other health care professionals, and the need to collaborate with others in caring for patients and promoting health • A commitment to provide care to patients who are unable to pay and to advocate for access to health care for members of underserved populations

41. “Novice” “Advanced learner” “Expert” 2. Developmental Terms: can identify absence of pulse can distinguish specific arrythmias can manage ventricular fibrillation

42. Dreyfus and Dreyfus • Novice • Advanced beginner • Competent performance • Proficient performance • Intuitive expert • Master students residents facutly Mind Over Machine(1986)

43. Developmental vs. Analytic • time-line, progression included levels of function • essential for multi-year training • although the terms (“novice”, “master”…) remain generic

44. Goals: ACGME “Competencies” • Medical Knowledge • Interpersonal & communication skills • Professionalism • Patient Care • Practice-based learning • System-based Practice What about the last three?? analytic? developmental?

45. A cube to encompass competence UME --> GME  CME school Medical Knowledge Patient Care Interpersonal skills Professionalism Practice-based learning/improv. System-based Practice hospital clinic

46. Resistance is futile

47. Educational Goals Curriculum Evaluation Feedback / Grading community Depends on expectations

48. 3. the “Synthetic”framework • “syn-thetic” -putting the learner back together • “K S A” are all required, integrated • terms are a bit less generic, more behavioral

49. “R.I.M.E. Scheme” • Reporter • Interpreter • Manager/Educator Pangaro, Academic Med, 1999

50. a framework to classify level of function • rudimentary reporting: “My patient has a fever, cough and a bad rash - it’s vesicular or pustular ….” • rudimentary interpreting: “I think it might be due to chicken pox or herpes.”