PEER ASSESSMENT (PA) OF MEDICAL PROFESSIONALISM BY MEDICAL STUDENTS

1. PEER ASSESSMENT (PA) OF MEDICAL PROFESSIONALISM BY MEDICAL STUDENTS AFMC PROFESSIONALISM RESOURCE GROUP MEETING A Keith W Brownell MD London – April 29, 2006

2. OVERVIEW • References • Background to My Interest in the Topic • General Comments About PA • Specific Comments on PA of Professionalism by Medical Students • Some Canadian Experiences • Discussion

3. DISCLOSURE/DISCLAIMER None of this material is original and most of it comes directly or indirectly from one of the key references which are listed on Slide 5

4. BACKGROUND TO INTEREST IN THIS TOPIC

5. KEY REFERENCES • John Norcini – Peer Assessment of Competence. Medical Education 2003:37:539-543. • Deirdre Lynch et al – Assessing Professionalism: A Review of the Literature. Medical Teacher 2004;26(4):366-373. • Carolyn Shue et al – Maximizing Participation in Peer Assessment of Professionalism: the Students Speak. Academic Medicine 2005;80(10 Suppl):S1-S5. • Elaine Daddefer et al – Peer Assessment of Professional Competence. Medical Education 2005;39:713-722. • Louise Arnold & David Stern – Content and Context of Peer Assessment in Measuring Medical Professionalism – Edited by DT Stern – Oxford University Press 2006

6. GENERAL COMMENTS ABOUT PA • The act of making judgments on the performance of one’s peers is ubiquitous and has formed the basis of the referral process in medicine and other professions for centuries. • Throughout this long history, peers have been deployed in a variety of ways to make judgments on the competence of their colleagues.

7. NATURE OF JUDGMENTS • Judgments about structured tasks versus global impressions. • Judgments about occurrence, quality or suitability.

8. JUDGMENTS ABOUT STRUCTURED TASKS VERSUS GLOBAL IMPRESSIONS • Global impressions judgments are more common than judgments about structured tasks. • Each type has strengths and weaknesses – so for example • Global impressions influenced more by halo effect. • Structured tasks are limited by the number of encounters being assessed.

9. JUDGMENTS ABOUT OCCURRENCE, QUALITY OR SUITABILITY • Occurrence can be assessed by check lists. • Quality of performance is more commonly assessed but limited by number of observations. • Is the performance suitable/satisfactory – this involves determining if the performance was of good quality and also good enough for the purpose of the evaluation. More difficult to do.

10. Technical /Cognitive Technical ability Basic science knowledge Clinical knowledge Judgment Problem solving Relationship/Non-cognitive Peer relations Patient relations Reliability Industry Personal appearance Reaction to pressure ASPECTS OF COMPETENCE ASSESSED

11. FACTORS INFLUENCING THE QUALITY OF PA • Reliability • Relationships • Stakes • Equivalence

12. FACTORS INFLUENCING THE QUALITY OF PA (I) Reliability • This is influenced by the • Number of relevant observations • Number of peers involved • Number of aspects of competence involved • Influenced little by wording of questions, number of points on rating scales, whether to describe all points on scale etc.

13. FACTORS INFLUENCING THE QUALITY OF PA (II) Relationships • Are the peers competing with each other? • Are the rating peers friends? • Are there financial relationships?

14. FACTORS INFLUENCING THE QUALITY OF PA (III) Stakes • Likely to be influenced by use to which it will be put – with high stakes then tendency to give higher ratings. This can be mitigated to some degree by ensuring anonymity of raters. • Ensure that peer is being asked to rate on quality of performance rather than suitability for something.

15. Equivalence This means being judged on same activities being judged by peers who all have the same degree of stringency (need to avoid hawks and doves) Problems can be minimized by Increasing the number of peers doing the assessment Providing the peers with clear criteria for making their judgments FACTORS INFLUENCING THE QUALITY OF PA (IV)

16. STEPS IN IMPLEMENTATION • State purpose of assessment, preferably in writing. • Assessment criteria must be developed and communicated to the participants. • Training should be provided to all participants. • The results of the assessment should be monitored throughout the implementation process. • Feedback should be provided to the participants.

17. NOW TO MEDICAL PROFESSIONALISM AND PEER ASSESSMENT OF MEDICAL STUDENTS

18. Why Do This? • Medical teachers have less time to spend in direct contact with students. • As a result opportunities for learners to demonstrate a wide variety of professional and unprofessional behaviors are more likely to occur among peers who spend more time with one another and work closely together as a team. • Professional behaviors like responsibility, effective communication, interpersonal respect, thoroughness and altruism have a direct impact on peers or reflect values that peers might be able to infer and observe in actions of their peers.

19. The medical student peers are individuals who have attained the same level of training or expertise, exercise no formal authority over each other, and share the same hierarchical status in an institution. • These non-hierarchical relationships can promote both authentic behavior and genuine feedback among peers while reducing the biasing influence of social desirability. • In PA, peers are asked to judge each others’ characteristics or behaviors relevant to an evaluation task.

20. PA particularly useful in situations in which • Peers are afforded a unique view of one another’s behavior • Peers are capable of accurately perceiving and interpreting one another’s behaviors • There is a need to improve the effectiveness of assessment of group members’ behavior • Peers not only observe different behaviors but they also provide a different perspective on the same behaviors observed by others.

21. Methods of PA (I) • Ratings – involve each member of a group rendering a judgment about other members on a specified set of behaviors, performances or characteristics using a scale. • Nominations – consist of group members naming a certain number of group members as the best along a particular performance dimension or quality. • Rankings – involve each group member ordering all other group members from best to worst on specified behavioral dimensions or characteristics.

22. Methods of PA (II) • Peer voting – assign learners a number of votes equal to the number of group members and directs learners to apportion their votes across the group members according to the degree to which each member exhibits a behavior or quality relevant to the evaluation. • Qualitative comments.

23. PSYCHOMETRIC CHARACTERISTICS OF DIFFERENT METHODS USED IN PA

24. RATINGS • Most common method used and most supported with evidence of reliability and validity. • Validity includes face, content, construct, concurrent and predictive validity. • Method of choice when specific information about each group member is desired. • Especially useful for formative assessments which provide feedback. • They are subject to problems of bias and may be less valid because they judge across the entire range of performances or qualities.

25. CONTEXT OF PA • The reaction of learners to PA and the procedures used for initiating, maintaining, and reporting peer assessments can affect the integrity of evaluations reported by peers. • The use to which PA results are put may influence the peers willingness to provide genuine evaluations. • Medical students seem to be resistant to the idea of PA.

26. PA MOST LIKELY TO SUCCEED IF STUDENTS KNOW • How the results of it will be used. • That it is anonymous. • That it is being done at an appropriate time. • It is mandatory. • It covers areas in which peers see themselves as particularly perceptive. Or in other words that the content reflects the ideas and the lives of those who will participate in the system.

27. PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (I) • School responsiveness to peer reports. • Behavioral standards on professionalism that Faculty consistently enforce. • Student groups whose members are amenable to teaching each other and exploring professionalism issues. • Close relationships between students and Faculty. • Approachable administrators who trust students. • Value placed on assessment aimed at learner’s improvement.

28. PA MOST LIKELY TO SUCCEED IF IT OCCURS IN THE CONTEXT OF A SUPPORTIVE ENVIRONMENT WHICH WOULD INCLUDE (II) • Education which explores the meaning of professionalism. • Specifies expectations for professional behavior. • Provides training for feedback on professionalism and conflict resolution. • Highlights the importance of PA. • Provides for faculty modeling of PA.

29. SURVEY • January – sent e-mail to everyone on the AFMC Professional Resource Group Mailing List. • No names listed from Francophone Schools, NOSM or Memorial on the AFMC mailing list thus no contacts were made with any of these schools. • Arranged telephone contacts with the people who responded.

30. CANADIAN EXPERIENCES WITH MEDICAL STUDENT PA • University of British Columbia – Niamh Kelly – PBL. • University of Alberta – Lorraine Breault – Interdisciplinary HSC Course. • University of Calgary – Janet Wright & Rod Crutcher – Well Physician Course. • University of Western Ontario – Peter Flanagan – PCL. • University of Ottawa – Walter Hendelman – Professionalism Program. • Others.

31. DISCUSSION • Is peer assessment a good thing to do? • If yes, why are we not doing more of it? • What could this Resource Group do to move PA forward in our schools? • Interest in making peer assessment of medical professionalism a focus of this resource group? • Other issues?