Principles of feedback delivery and the role of self- assessment

1. Principles of feedback delivery and therole of self-assessment Kevin W. Eva Department of Clinical Epidemiology and Biostatistics Program for Educational Research and Development McMaster University

2. Principles of feedback delivery • There are no universal principles

3. “There is no “best” type of formative feedback for all learners and all learning outcomes” Shute (2008)

4. Principles of feedback delivery • There are no universal principles • Self-assessment is insufficient

5. The Rhetoric • Almost every article on self-assessment begins with the same basic sentence: “The ability to self-assess is a cornerstone of professional self-regulation”

6. The self-regulating professional • Reflect regularly on daily practice • Self-assess gaps in knowledge or skill • Seek opportunities to redress gaps • Invest energy to learn (or relearn) • Incorporate knowledge into practice • Repeat (Handfield-Jones, et al., 2002)

7. Eva, et al. (2004)

8. The self-assessment literature • Hundreds of articles • Many literature reviews • One conclusion Self-assessment ability is generally poor

9. Boehler, Rogers, et al. (2006)

10. “In a majority of relevant studies physicians do not appear to accurately self-assess” Davis, et al. (2006)

11. The self-assessment literature • Either the conclusions of hundreds of studies using multiple methodologies are all wrong or a critical premise underlying the concept of self-regulation in the professions is unsupportable

12. If you don’t think you’re susceptible, you’re not alone … … Pronin, Lin, Ross (2002)

13. “Evolution deserves the Microsoft Windows Award for installing these mental processes in every one of us without asking permission.” (Gilbert, 2006)

14. Principles of feedback delivery • There are no universal principles • Self-assessment is insufficient • Self-assessment is vital

15. “It is impossible for a man to learn what he thinks he already knows” Epictetus (circa 100AD)

16. Predicting generation of learning goals (Overall R2=.45; Eva, et al., In press)

17. “Physician agreement with colleagues’ scores correlated positively with overall mean scores from colleagues (R=.48)” Sargeant et al. (2003)

18. The ironic nature of self-assessment • We can’t trust it … but we should encourage it • We need to influence it … without changing it

19. Self Efficacy • Influenced by feedback, but also influences future performance • Not merely a passive reflection of performance, but part of a self-fulfilling prophecy that affects performance (see Dweck, 1999)

20. Implications for self-regulation • “Rose colored glasses” approach to reflection understandable and necessary • Important for ability to function and succeed • Can never create a “good self-assessor” • And probably shouldn’t try

21. Principles of feedback delivery • There are no universal principles • Self-assessment is insufficient • Self-assessment is vital • The goal is to enable discovery

22. % of physicians for whom it is difficult/impossible to generate practice data about… (Audet, et al., 2005)

23. % of physicians who have access to any quality of care data (Audet, et al., 2005)

24. The need for difficulties • Learners tend to request feedback after successful experiences (Chiviacowsky and Wulf, 2002) • Learning strategies perceived as beneficial are often sub-optimal (Simon and Bjork, 2001) • Proficiency tends to be inferred from the rate of learning rather than performance (Eva, 2009)

25. Jowett, et al. (2007) Learner Material

26. Test-enhanced learning See Larsen, Butler, and Roediger (2008)

27. “The phrase self-assessment should not imply an isolated or individualistic activity; it should commonly involve peers, teachers, and other sources of information” (Boud, 1999)

28. “Education is learning what you didn’t even know you didn’t know” (Boorstin, 1914)

29. Principles of feedback delivery • There are no universal principles • Self-assessment is insufficient • Self-assessment is vital • The goal is to enable discovery • Numbers can be harmful

30. “Feedback interventions that draw attention to the self via normative cues are largely ineffective” (Kluger and DeNisi, 1998)

31. “Studies that augmented feedback with consultation produced substantially larger differences, but other methodological variations had little effect” (Marsh and Roche, 1997)

32. Principles of feedback delivery • There are no universal principles • Self-assessment is insufficient • Self-assessment is vital • The goal is to enable discovery • Numbers can be harmful • Credibility is likely to be the key

33. Credibility “With your clinical partner … they know your strengths and weaknesses and they’re probably going to give you quite useful feedback. Maybe a consultant watched you one time. They might not have the whole picture of what you’re capable of and what you normally do” (Sargeant, et al., 2010)

34. Credibility Of relationships: Engagement and awareness Of the process: Validity and authenticity Of intent: Beneficence and non-maleficence (Sargeant, et al., 2010)

36. Thanks evakw@mcmaster.ca

37. Kruger and Dunning (1999)

38. Kruger and Dunning (1999)

39. The self-assessment literature(as summarized by Eva and Regehr, 2008) • Self-assessment broadly conceived of as a prequel to self-directed learning activities aimed at performance improvements • With respect to patient safety, we’re much more concerned about situation specific awareness (i.e., “reflection in practice”) • Continuous monitoring of whether or not one has sufficient skills to effectively solve the current problem

40. Self-monitoring Knowing when to defer • Participants clearly knew when they were likely to respond in error (Eva and Regehr, 2007)

41. Self-monitoring (Eva and Regehr, 2007)

42. Self-monitoring • Self-assessment (as overall judgment of ability) is importantly different from self-awareness (monitoring) • Difficulty with self-assessment is not that we can’t judge our performance, but that we don’t aggregate well over events