Development and Implementation of a Log-based Assessment Tool for Clinical Clerkship

1. Development and Implementation of a Log-based Assessment Tool for Clinical Clerkship Susan Chamberlain, MD, FRCSC Phil Hahn, MSc Dept of Obstetrics & Gynaecology Health Sciences Education Rounds May 21, 2009

2. Objectives • Describe log system linked to assessment and its rationale • Present results of survey of Meds2009 regarding their attitudes towards their clinical logs (quantitative and qualitative data)

3. Background Problem # 1: ED-2 requires students in clerkship to log clinical experiences but student participation is low • Students are not altruistic • Logs are incomplete and inaccurate • If results will be used to direct courses, they must be accurate • Students will only provide accurate information, if there is clear individual benefit

4. Background Problem # 2: Ob/Gyn end of rotation SAQ exam • Out of date • Time consuming to mark and maintain • Inconsistency in marking

5. Background • January 2006 (Meds 2007) – new system of evaluation linked to clinical log started • Each logged encounter corresponded to LMCC clinical presentation • Each covered by clerkship seminar • Students logged patient encounters, seminars attended • Exam question chosen based on clinical log (i.e. students only examined in SOE on what they had seen)

6. Background • MCQ lengthened • SAQ replaced by Structured Oral examination (SOE) • 50 mark questions focusing on data gathering (history, physical and choice of investigation, plus basic management choices) • Each exam question written by attending staff who gives the seminar (all reviewed/edited by course chair for consistency) • Student answers 2 questions in 30 minute exam

7. Survey on Student Attitudes March 2008 • Survey of class of meds 2009, blocks 2-8 on use of logs, attitudes • Student asked to complete online questionnaire after 2 clerkship rotations: ob/gyn and surgery • Confidentiality of responses assured by code to identify responses • 74 student surveyed

8. Survey Results • 34 Clerks filled out questionnaire after one of the rotations (OB or Surgery) • 19 Clerks filled out questionnaire after both rotations (paired data) • Responses after first rotation (either OB or Surgery) were included in the analysis

9. Survey Results Codes One duplicate (0023f91973) (Block 2 OB and Block 3 Surgery) One with no match (a0f3d5c663) These two responses were not included in analysis

10. Survey Results • 53 Clerks • Two independent groups • 29 OB rotation • 24 Surgery rotation

11. Q1. I completed my log: • After each patient • Daily • 1-2 times/week • Less than once/week

12. Q1. I completed my log: * One response (both 1-2 times per week and less than once per week) not included Values = n (%)

13. Q1. I completed my log: For analysis, “daily” and “1-2 times per week” categories were pooled. Values = n (%)

14. Q1. I completed my log: Values = n (%) P = 0.011, Fisher’s exact test (statistically significant) Clerks logged more frequently during the OB rotation.

15. Q2. I found logging useful to my learning. Yes No

16. Q2. I found logging useful to my learning. OB: 20.7% (6/29) Surgery: 12.5% (3/24) Difference = 8.2% P = 0.487, Fisher’s exact test

17. Q3. I logged some clinical encounters that I did not see. Yes No

18. Q3. I logged some clinical encounters that I did not see. OB: 10.3% (3/29) Surgery: 4.2% (1/24) Difference = 6.1% P = 0.617, Fisher’s exact test

19. Q4. I chose not to log some clinical problems that I did see. Yes No

20. Q4. I chose not to log some clinical problems that I did see. OB: 62.1% (18/29) Surgery: 62.5% (15/24) Difference = 0.4% P = 1.000, Fisher’s exact test

21. Q5. After logging clinical encounter, I did further reading on the topic ___% of the time

22. Q5: After logging clinical encounter, I did further reading on the topic: % of Clerks OB Rotation (n=28), 1 missing Median (Middle) Response = 15% of the time Response: % of the time.

23. Q5: After logging clinical encounter, I did further reading on the topic: % of Clerks Surgery Rotation (n=23), 1 missing Median Response = 10% of the time Response: % of the time.

24. Q5. After logging clinical encounter, I did further reading on the topic: • Median (% of the time) • OB: 15% versus Surgery: 10% • P = 0.453, Mann-Whitney U test (nonparametric version of the t test)

25. Q6: Average time (minutes) spent reading on an clinical encounter

26. Q6: Average time (minutes) spent reading on an clinical encounter OB Rotation (n=28), 1 missing % of Clerks Median = 30 minutes Time (minutes)

27. Q6: Average time (minutes) spent reading on an clinical encounter Surgery Rotation (n=22), 2 missing % of Clerks Median = 30 minutes Time (minutes)

28. Q6. Average time (minutes) spent reading on an clinical encounter • Median (minutes) • OB: 30 versus Surgery: 30 • P = 0.935, Mann-Whitney U test (nonparametric version of the t test)

29. Q7. Logging prompted me to read more about patients that I had seen. Yes No

30. Q7. Logging prompted me to read more about patients that I had seen. OB: 6.9% (2/29) Surgery: 12.5% (3/24) Difference = 5.6% P = 0.649, Fisher’s exact test

31. Q8. Logging clinical encounters helped focus my studying. Yes No

32. Q8. Logging clinical encounters helped focus my studying. OB: 20.7% (6/29) Surgery: 4.2% (1/24) Difference = 16.5% P = 0.112, Fisher’s exact test

33. Summary Fisher’s exact test

34. Summary Mann-Whitney U test nonparametric version of the t test Values are medians (5th, 95th percentiles) Q6. One missing value in each group Q7. One missing value in OB group and two missing values in surgery group.

35. Conclusions (Quantitative data) • There is no difference with regards to study use of the log or students’ perception of usefulness of logs as an educational tool between these 2 rotations • Students on the ob/gyn rotation logged more frequently • A minority of students find that logging is useful to their learning

36. Conclusions – accuracy of logs • Up to 10% of students logged clinical problems that they did not see • 62% of students are not logging the clinical problems that they are seeing

37. Conclusions – log as study tool • Few students use the log as an impetus to read around patient encounters • Few students find the log helpful as a focus for their study

38. Survey Results – qualitative data Q9: Please comment on how clinical encounter logs may have contributed to your education in either a positive or negative way.

39. Results – technical/communication issues • Students find the interface difficult and are frustrated by the limited choices, what ‘counts’ as a clinical encounter • “too many drop-down menus” • “topics were not arranged in a useful way” • “cumbersome” • “... told by staff that you don’t have to log everything, then get in trouble when not everything is logged.”

40. Results – logging is a ‘make-work project’ • Students consider the log ‘an administrative task which has no bearing’ on learning • “felt like a tedious waste of time” • “..waste of time that could have been used studying..” • …

41. Results – Honesty/accuracy of the logs • Unclear thresholds for logging of clinical activity • “sometimes difficult to know what counts as having ‘seen’ a clinical problem” • “…I logged...only those patients that I actually assessed…provides the greatest learning”

42. Results – honesty/accuracy • Inaccurate recollection • “I was inconsistent in filling logs out, sometimes forgetting for up to a week” • “I was fairly reliable about logging for the first 5 blocks, then completely forgot about it for the last 3”

43. Results – honesty/accuracy • Students logged encounters they did not see • “..often completing logs at the conclusion of a rotation by simply making up cases that would allow me to cover rotation objectives” • “ does not ensure anything as it can easily be faked” • “..students may be tempted to log encounters they have not experienced to avoid such emails”

44. Results – honesty/accuracy • Students did not log encounters they did in fact see • “I did choose to not log a specific encounter because I did not want to be asked the question on my oral exam as I did not have the chance to fully read around the topic”

45. Results – study habits • Some students found the log useful for studying: • “It does cause me to reflect on particular cases but fitting this extra task in my daily schedule feels like adding one more juggling ball to a performance that exists on the edge of completely falling apart” • “It probably did not increase my reading of topics I did encounter, but would likely make me focus on those topics I hadn’t encountered”

46. Results – study habits • Most students did not find the log useful for studying: • “I read about things that I did not understand as they came up, rather than after I had logged them.” • “Logging was not a motivation for me to read more, seeing my patients was.”

47. Results – Q9 • “.. I haven’t found it helpful for identifying areas of deficiencies. For example I went through 6 weeks of surgery (incl urol) without having a single opportunity to put in a foley catheter” • “… after the mid-way meeting and outside of ob/gyn, no one else seemed to give a toss about the log, so I didn’t either” • “these .. would be more beneficial if they were emphasized by the surg/obsgyn teams during clerk rotations, with some minor teaching involved…

48. Results – positive comments • Ensured rotation objectives met: • “they direct you to see out certain clinical opportunities that you haven’t had yet” • “Did not directly affect my learning for the block, but did ensure that I had seen all the objectives laid out for the block”

49. Survey Results – qualitative data Q10: With respect to the OB/GYN rotation, how did knowing that the topics for your final oral exam were selected after reviewing your clinical encounter log: • A: Influence your logging habits? • B: Contribute to your overall learning experience?

50. Survey Results – qualitative data Q10: With respect to the OB/GYN rotation, how did knowing that the topics for your final oral exam were selected after reviewing your clinical encounter log: • A: Influence your logging habits?