Creative Commons License

1. Creative Commons License Attribution-NonCommercial-ShareAlike 2.0 • You are free: • to copy, distribute, and display this presentation, and/or • to make derivative works • Under the following conditions: • Attribution. You must give the original authors credit. • Noncommercial. You may not use this work for commercial purposes. • Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. • See http://creativecommons.org/licenses/by-nc-sa/2.0/ for full license.

2. Portfolios – Concepts and Realities ePORT Meeting April 2, 2008 Lee A. Learman, MD, PhD Director of Curricular Affairs, Office of GME

3. Portfolios for Assessment • Purposeful collections of evidence used by students to document and reflect on learning outcomes over time. • Potential uses: • Evidence database: by program vs. learner • Formative: coaching, feedback • Summative: decisions regarding progress Dannefer EF, Henson LC. Acad Med 2007;82:403.

4. Why Portfolios? • Exciting and innovative tool for resident learning and development • Already used in K-12, higher education, and multiple professions • Relieving burden while increasing accuracy for program directors • “added” or “lifted” - “eye of the beholder” • Building a community of practice for GME • Within and across specialties to raise the bar http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

5. Why Portfolios? • Needs identified in the literature: • Focus on complex tasks, integrated competencies • More feedback needed as learning objectives become defined and measured more accurately • Context-dependence: sample multiple sources • New ways to analyze, summarize all the data • Portfolios can meet these needs Dannefer EF, Henson LC. Acad Med 2007;82:403.

6. Portfolio Approach at CCLCM Dannefer EF, Henson LC. Acad Med 2007;82:403.

7. Ingredients for Success • Time for reflection and mentorship • Separation of formative and summative • Students select evidence of learning • Essays required to aid reflection on integration of competencies • Rigorous measurement standards for summative assessments (fair, valid, reliable) Dannefer EF, Henson LC. Acad Med 2007;82:403.

8. ACGME Portfolio Management Tool “An interactive web-based development tool that residents can use throughout their residencies to record and organize their learning and to reflect and receive feedback on their skills as physicians, building evidence that allows them to chart their own progress over time.” http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

9. ACGME Portfolio Management Tool • “First and foremost . . .a learning tool for residents that enables them to”: • track their experiences • self-reflect on those experiences • share their insights with mentors, • receive real-time formal feedback • A repository for resident work products and professional documents meeting the needs of many groups including licensing and certification boards http://www.acgme.org/acWebsite/portfolio/cbpac_faq.pdf

10. Portfolio Functions - ACGME • Growth Model (formative) – tracks learner development over time • Showcase Model (summative) – snapshot demonstrating achievement of identified outcomes as for a grade, promotion, or graduation • Hybrid – supports both purposes http://www.acgme.org/acWebsite/portfolio/cbpac_memo.pdf

11. ACGME Outcome Project • Portfolios – What Might We Expect from the ACGME? Tina Foster, MD, MPH

12. So What Does All That Mean? • One place for record of resident experience: evaluations, case logs, other work. Portions of record can move forward with resident after completion of training. • With a better record of what residents have done and how they have demonstrated competency, PDs can more easily assess their program outcomes and DIOs their institutional outcomes. RRCs can benefit from aggregated data; aspects of portfolio could replace some of PIF narrative

27. Known Challenges • No added burden – or if burden is added here, where do we take it away? • If we have to add a little burden…how do we know it’s worth it? What’s the added value? • Link to post-residency activities will be important - -this is a “lifelong” portfolio

28. Self-Reflection & Portfolios Evidence Warehouse Learners Mentors Instructions on reflection A method for assessing reflection

29. Reflection at UCSF: Instructions • Describe the situation that “taught you the most” about [specific competency x 6] • Describe challenge(s) faced, strategies used • Describe sources of feedback (people, data) • Relate the situation to previous similar ones • Include details to illustrate challenges you faced and lessons you learned • List conclusions re: strengths, opportunities for improvement, and use examples to justify conclusions Learman LA, Autry AM, Pliska L, O’Sullivan PS. WGEA Meeting 2006.

30. Changing the System Sent: Wed 1/10/2007 10:04 PM Subject: Proposal for High Risk OB Clinic Hello all!As many of you know, Thursday mornings pose a challenge to even the most efficient of residents.  Juan and I were discussing the following minor changes to resident roles and Thursday mornings High Risk OB clinic that could potentially make a major difference…Please look over these proposed changes. I would love your feedback – most of this was born from the question of how to make M&M a more consistently educational experience for all (i.e., how can the OB chief and R2 get to OB M&M on time?) -1)      All residents (including OB team) arrive at 5M on Thursdays at 8:00 am to start prepping charts; attending and fellow arrive at 8:30 to hear presentations of prepped charts. Conference should be finished by 9 am. If chart prep not complete and patients present at 9 am, at least 2 residents should leave conference to start seeing patients.2)      Night float chief helps to prep some charts on Wed night for Thursday morning so that conference can be finished by 9 am3)      OB Chief must leave clinic at noon to start M&M - supported by clinic attending4)      OB R2 next to leave, ideally at noon as well5)      Gyn R2 and Clinic Chief last to leave clinic, to cover those 12:30 must-see patients (Gyn R1 covering pager and consults if Gyn R2 needed after 12)6)      If >8 patients remain to be seen at 12:00, then OB R2 stays with the rest of the residents while OB Chief leads M&M alone. If needed, other attendings (mobilized by Dr.V) help to see patientsWithout compromising patient care for education, I'm hoping this may help both clinic flow and M&M utility. Thanks for your time!

31. Reflection on Systems-based Practice (1) Describe the setting and context including who was present. Setting: San Francisco General Hospital, Thursday morning moderate risk obstetrics clinic. Those involved iare residents, faculty staffing the clinic, clinic staff and clinic flow nurse (2) What challenges did you face in practicing cost-effective healthcare or surmounting systemic barriers to optimal care? The challenge I hoped to address was optimizing the care of patients with complex obstetric problems in the outpatient setting. The major obstacles have been in existence for years: many patients requiring both clinical attention and ancillary services, within a limited time. One additional challenge is finishing the clinic at a time that would allow some or all of the residents to attend M&M conference, a major learning opportunity. (3) Describe what efforts you made to surmount the challenges. What past experiences did you bring to this situation? In the past, there has been a common feeling that “the clinic is just that way – it’s hopeless to try and change it.” I tried to separate myself from that because I wanted people to want to change! In conjunction with one of our faculty members, I came up with ideas on how residents could contribute time that did not risk violating duty hour regulations in order to improve clinic flow. For example, the night float team helping to prepare charts, residents arriving a half hour earlier to finish preparing charts, and then presenting to attendings two at once so that we could start seeing patients as soon as they were placed in rooms . . . (4) How did you obtain feedback and from whom? For the last two weeks, our goal of getting at least two residents to the M&M conference has been successful, but the rest of the residents are not having the opportunity to benefit from this educational conference as I had hoped. I have been asking the faculty who staff the clinic, residents, and the clinic flow nurse how the system has been working. The reviews have been mixed, and since it has been only been implemented for a short time, it’s difficult to determine whether or not this system change is actually beneficial. (5) List what conclusions you drew from situation regarding your strengths and opportunities for improvement, and use examples from the situation to justify your conclusions. I found that motivating people towards change was simple and that those involved were willing to attempt to motivate change themselves, once the activation energy was there. I felt that communicating the importance of this change was one of my strengths. However, putting emotional investment in it means that I have been disappointed that in the first two weeks it hasn’t seemed to work. (6) What changes, if any, do you plan to make if you face a similar situation in the future? I plan to encourage future residents to continue with this system and possibly find even more adjustments to improve its efficiency

32. Bridge from Data to Discovery Evidence Warehouse Reflection is scored as evidence of a specific competency or PBLI (summative) Learner receives, instructions and selects experience Discusses reflection with mentor (formative) Reflects and summarizes