“Criterion Based” vs. “Norm-Based” Evaluation

1. “Criterion Based” vs. “Norm-Based” Evaluation David L Gaspar MD October 18, 2008 UCD School of Medicine

2. Evaluation of Students Goals-This session is designed to enable participants to: 1. Understand the definition and purpose of evaluation 2. Develop a framework to make evaluation efficient and useful 3. Understand and apply criterion-based evaluation – the “R.I.M.E.” method UCD School of Medicine

3. Evaluation Basics Definition Evaluation is the process of making judgments based on factual information and observations in order to rate, rank, or assess an individual’s status at a given point. Purposes of Evaluation 1. Summarize performance at a given point in time. 2. Provide information for planning future educational experiences. 3. Communicate summary information to other parties. UCD School of Medicine

4. Framework for Planning Clerkship Evaluation Before Clerkship 1. Understand medical school expectations and “forms”. 2. Review clerkship goals and objectives. During Clerkship 1. Gather information from multiple sources & provide feedback. 3. Use systematic method of recording. End of Clerkship 1. Prepare for final evaluation. 2. Schedule and conduct summary meeting. 3. Complete and submit final evaluation. UCD School of Medicine

5. Tips to Facilitate Evaluation • Dictate or write a student progress note. (end of day, when procedures occur-behaviours) • Use student log books. • Review student’s written record. (copy progress notes at the beginning, middle, and end of clerkship) • Use student self-assessment forms. • Use computer printout of student’s patient profile. UCD School of Medicine

6. Evaluate… End of Clerkship 1. Review evaluation criteria and form. 2. Review student performance data. • Schedule and conduct final evaluation session with student. • Have some “backbone” 5. Complete and return the evaluation form promptly. (Keep a Copy!!!) UCD School of Medicine

7. “Norm”-Based vs. Criterion-Based Any ideas???? UCD School of Medicine

8. “Norm”-Based vs. Criterion-Based NORM-BASED EVALUATION • Norm-based assessment compares individuals’ performances to one another, resulting in a ranking of individuals. Medical Knowledge UCD School of Medicine

9. “Norm”-Based vs. Criterion-Based NORM-BASED EVALUATION • This traditional form of assessment has been widely used to select individuals for further education or for jobs and is what most students, residents, fellows and faculty have been exposed to throughout their training UCD School of Medicine

10. “Norm”-Based vs. Criterion-Based NORM-BASED EVALUATION • Since norm-based assessment provides no information about the observed quality of the performance, it isn’t useful for providing feedback or improving performance. • The rater’s task ends up distinguishing one individual’s performance from another’s. UCD School of Medicine

11. “Norm”-Based vs. Criterion-Based NORM-BASED EVALUATION • Raters will vary in their assessments depending on their experience with using a norm-based scale, and their experience with the group being evaluated UCD School of Medicine

12. Dr. Thomas, who has 15 years of experience working with and evaluating students, is due to evaluate John, his student over the last 2 weeks. He has observed John performing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. Dr. Thomas has a clear expectation based on this experience for what an “honors student” is. He believes that John is a very good student with great enthusiasm, but not quite functioning at an honors level and will therefore complete his evaluation in a manner that reflects what he thinks will result in a “high pass” grade- on a 6-point scale. mostly 5s and some 6s. 1= poor performance and 6=excellent performance, NORM-BASED EVALUATION UCD School of Medicine

13. Dr. Thomas’ evaluation: 1= poor performance and 6=excellent performance, NORM-BASED EVALUATION UCD School of Medicine

14. The next attending to work with John, Dr. Smith, has just completed residency. She too has been working with John for 2 weeks and has observed him doing a brief history and physical examination, has heard daily oral presentations, and has interacted with him on multiple occasions to assess his fund of knowledge. She is an enthusiastic teacher and believes that students should get honors if they are active participants in rounds, enthusiastic about learning and generally helpful. She went to a medical school where most students got honors if they did these things and she thinks this is a good policy. She has heard from the medical students and residents that she has worked with that to get honors, she must give John all 6s on a 6 -point scale 1= poor performance and 6=excellent performance, NORM-BASED EVALUATION UCD School of Medicine

15. Dr. Thomas’ evaluation: 1= poor performance and 6=excellent performance, NORM-BASED EVALUATION UCD School of Medicine

16. What grade does John deserve? More importantly, what is John’s level of competence? NORM-BASED EVALUATION UCD School of Medicine

17. Answer: Hard to tell…..This example provides insight into some of the challenges of norm -based assessment. Now let’s learn about an alternative means of assessment. NORM-BASED EVALUATION UCD School of Medicine

18. How do we evaluate the learner with a criterion-based system? • There have been a variety of methods used over the past several decades • In 1999 L. Pangaro developed RIME – a new tool for evaluating the level of your learner – more specifically for medical students UCD School of Medicine

19. RIME • Reporter • Interpreter • Manager • Educator • Before RIME – Observer/Pre-Reporter (pre-clinical) UCD School of Medicine

20. RIME • Reporter - Consistently good in interpersonal skills, reliably obtains and communicates clinical findings. Written and spoken presentations are clear and organized. • Interpreter – Able to prioritize and analyze patient problems, develop a differential diagnosis and next steps in workup. Data is not just data (reporter), but has a clinical significance. UCD School of Medicine

21. RIME • Manager – Consistently proposes reasonable diagnostic and therapeutic options, incorporating patient preferences. They can take their knowledge, integrate it with the findings for a given patient and form a plan that prioritizes within and amongst the problem list. • Educator – Consistent level of knowledge of current medical evidence; demonstrates self-directed learning (practice based improvement and learning) and contributes to others’ education. UCD School of Medicine

22. RIME Changes with Time • You can use RIME to track the progress/growth of a student over the course of their clinical years or within a given year. • Early MS3 – Should report well and begin to interpret. • Late MS3 – Should be a good interpreter • Sub-I should have the above and begin to manage patients appropriately. • Interns should become astute managers • R2s and beyond – progressive depth of management and becoming educators UCD School of Medicine

23. RIME: Criterion-based evaluation • Criterion -based assessment begins with an established set of criteria for successful performance and compares an individual’s performance to these criteria. UCD School of Medicine

24. RIME: Criterion-based evaluation • Rating an individual’s performance against the criteria provides feedback on the quality of performance. Multiple individuals may successfully meet the criteria as one individual’s performance is not constrained by another’s performance. Agreement between raters is high as the standard for comparison is stable and based on observable behaviors. In other words, the rater is responsible for documenting what the specific skills of the individual are based on their observed performance. They are NOT responsible for comparing them to anyone else, or for trying to determine a grade. UCD School of Medicine

25. RIME: Criterion-based evaluation In the scenario just described, both Dr. Smith and Dr. Thomas have observed John’s oral presentations. When they are asked to complete the evaluation of John they are asked to rate his abilities based on observable behaviors related to his oral presentations. John’s oral presentations are accurate and objective. But, they sometimes contain extraneous information and important information gets lost in the presentation. Please complete the evaluation below as if you were Dr. Thomas, with 15 years of experience teaching medical student: UCD School of Medicine

26. RIME: Criterion-based evaluation Would you evaluate the student any differently if you were Dr. Smith? UCD School of Medicine

27. RIME: Criterion-based evaluation John is able to derive a rudimentary differential diagnosis and assessment for patients presenting with shortness of breath and chest pain with only minor assistance from an intern or resident. But, he has no idea how to approach the presenting problems of acute renal failure and hyponatremia. For each of these latter clinical situations, the intern needed to completely develop the differential diagnosis and teach it to John, who was then able to repeat it at attending rounds. He is only able to develop management plans with significant help from the intern. Each of these 4 conditions is common on your rotation and required clinical entities for the clerkship. How would you rate him on the following scale? UCD School of Medicine

28. RIME: Criterion-based evaluation John is able to derive a rudimentary differential diagnosis and assessment for patients presenting with shortness of breath and chest pain with only minor assistance from an intern or resident. But, he has no idea how to approach the presenting problems of acute renal failure and hyponatremia. For each of these latter clinical situations, the intern needed to completely develop the differential diagnosis and teach it to John, who was then able to repeat it at attending rounds. He is only able to develop management plans with significant help from the intern. Each of these 4 conditions is common on your rotation and required clinical entities for the clerkship. How would you rate him on the following scale? UCD School of Medicine

29. And Then Its in My Court… UCD School of Medicine