1 / 51

Observable Practice Activities (OPAs)

Observable Practice Activities (OPAs). Content-Based (COPA). Process-Based (POPA). Mid/End of Rotation Feedback (formative) Entrustment (1-5). Mapping. Mapping. Curricular Milestones. End-of-training EPAs. Reporting Milestones. External Reporting NAS report.

chika
Download Presentation

Observable Practice Activities (OPAs)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Observable Practice Activities (OPAs) Content-Based (COPA) Process-Based (POPA) Mid/End of Rotation Feedback (formative) Entrustment (1-5) Mapping Mapping Curricular Milestones End-of-training EPAs Reporting Milestones External Reporting NAS report Longitudinal Assessment (summative) Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  2. Mapping • We mapped with the following in mind: • when assessing a trainee on a given OPA, would the evaluator feel as if they were also assessing the mapped milestones? • We removed all soft calls Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  3. OPA: Minimize unfamiliar terms during patient encounters Milestones ICS-A2 - Effectively use verbal and nonverbal skills to create rapport with patients/families ICS-D1 - Deliver appropriate, succinct, hypothesis-driven oral presentations SBP-B2 - Work effectively as a member within the inter-professional team to ensure safe patient care

  4. GIM PGY-1 Wards

  5. COPA: Identify causes of delirium Reporting Milestones PC-1 - Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s). MK-1   Clinical knowledge

  6. COPA: Manage exacerbations of obstructive lung disease Reporting Milestones PC-2   Develops and achieves comprehensive management plan for each patient. PC-3   Manages patients with progressive responsibility and independence. MK-1   Clinical knowledge MK-2   Knowledge of diagnostic testing and procedures

  7. COPA: Initiate cost-effective workup for anemia Reporting Milestones PC-1   Gathers and synthesizes essential and accurate information to define each patient’s clinical problem(s) MK-1   Clinical knowledge MK-2   Knowledge of diagnostic testing and procedures SBP-3 Identifies forces that impact the cost of health care, and advocates for, and practices cost-effective care

  8. COPA: Assess and treat pain Reporting Milestones PC-2 Develops and achieves comprehensive management plan for each patient PC-3 Manages patients with progressive responsibility and independence MK-1   Clinical knowledge PROF-3 Responds to each patient’s unique characteristics and needs

  9. POPA: Demonstrate shared decision-making with the patient Reporting Milestones PC-2 Develops and achieves comprehensive management plan for each patient PROF-1 - Has professional and respectful interactions with patients, caregivers and members of the interprofessional team (e.g. peers, consultants, nursing, ancillary professionals and support personnel) PROF-3 Responds to each patient’s unique characteristics and needs ICS-1 - Communicates effectively with patients and caregivers.

  10. POPA: Use feedback to improve performance Reporting Milestones PBLI-3 - Learns and improves via feedback PROF-4 - Exhibits integrity and ethical behavior in professional conduct.

  11. Mapping OPAs to Milestones Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  12. Mapping OPAs to Milestones Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  13. Mapping OPAs to Milestones Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  14. Mapping OPAs to Milestones Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  15. Mapping OPAs to Milestones Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  16. Patient Care Patient Care Traditional Competency-Based Evaluations Manage Vent Read EKG Diagnose Hepatitis Titrate Insulin OPA Based Evaluations

  17. Titrate Insulin Read EKG Manage Vent Diagnose Hepatitis The Milestones are the apples to be compared

  18. Approximately 3000 Assessments

  19. Reporting Milestones Resident A Resident B Resident A Resident B Resident C Resident C

  20. Reporting Milestones Resident E Resident D

  21. Reporting Milestones Resident F Resident F

  22. 6 nurses

  23. Reporting Milestones (ACGME)

  24. Example: PGY-3 30 Month NAS Report 1 1.5 2 2.5 3 3.5 4 4.5 5

  25. NAS report Jul-Dec 2013

  26. Single PGY-1 - OPAs from Jul 1st to Feb 28th Entire Residency – All OPAs from Jul 1st to Feb 28th

  27. Trend line of all PGY-1 Milestone Assessments By Month n = 105721 Standard Deviation of all PGY-1 Milestones Assessments by Month

  28. Validity? • Validity – how well one can trust the results of a test as interpreted for a specific purpose1 • Not a property of the instrument, but of interpretation 1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422

  29. Validity? • Content: do instrument items represent the construct? • Response Process: are the construct and the thought processes of observers the same? • Reliability – internal consistency, temporal stability, inter-rater reliability, generalizability • Consequences: do scores really make a difference? • Relations to other variables 1Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med. 2006 Feb;119(2):166.e7-16. Review. PubMed PMID: 16443422

  30. R3 Class Milestone Average Jul 1st to Feb 1st Selected to be Chief Resident Had issues brought before competency committee Patient Care Communication Professionalism

  31. 2012 2013

  32. Research Directions • Study the progression of assessment entrustment decisions made over time for internal medicine residents by faculty physicians, peer physicians, and allied health providers • Identify assessment biases present in these entrustment decisions, including bias associated with age, sex, resident year of training, attending experience level, or job description of assessor • Determine validity of assessment entrustment decisions made over time for internal medicine residents in relation to other variables such as standardized test scores, and other clinical assessments

  33. Reporting Milestones Narratives OPAs Level Of Entrustment 1 2 3 4 5 Rides in Driveway Rides Bicycle Safely Falls off Bike Rides 1000 feet without Stopping Rides on Sidewalk Rides on Quiet Street Level Of Entrustment 1 2 3 4 5 Rides on Quiet Road Rides in Traffic Rides in Rush Hour Traffic Rides to work on time Rides Tour de France Rides in Triathlon

  34. Reporting Milestones/EPAs Narratives OPAs Level Of Entrustment 1 2 3 4 5 Develops and achieves comprehensive management plan for each patient. (PC2) Care plans are consistently inappropriate or inaccurate Titrate cardiac medications Manage pancreatitis Inconsistently develops an appropriate care plan Titrate insulin based on glucose readings Adjust short and long acting narcotics for cancer and sickle cell patients Consistently develops appropriate care plan Level Of Entrustment 1 2 3 4 5 Appropriately modifies care plans based on patient’s clinical course, additional data, and patient preferences Manage parathyroid abnormalities in the renal patient Deliver appropriate goal-directed therapy for severe sepsis Role models and teaches complex and patient-centered care

  35. Observable Practice Activities (OPAs) Content-Based (COPA) Process-Based (POPA) Mid/End of Rotation Feedback (formative) Entrustment (1-5) Mapping Mapping Curricular Milestones End-of-training EPAs Reporting Milestones External Reporting NAS report Longitudinal Assessment (summative) Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, Lee CK, Lee D, Wood S, Fichtenbaum CJ, Schauer D, Munyon R, Mueller C. Entrustment and Mapping of Observable Practice Activities for Resident Assessment. J Gen Intern Med. 2014 Feb 21. [Epub ahead of print] PubMed PMID: 24557518

  36. TEAM EFFORT ResidencyProgram Staff Dell Diers Warm Mathis Fichtenbaum Mueller O’Toole Pai Schauer Tolentino Lee Ashbrook Held Lee Wood Wexler ResidencyEducationCoordinators Kamath Schoch Lenz Khan Robertson Houk Falciglia Chaudhary Cohen

  37. Discussion questions from the survey • Is it possible to have a generic map that can be used for both Medicine and Peds? • You could design an excel solution which Eric initially did before medhub figured out how to do it for us • Other ideas?

  38. Discussion questions from the survey • Specific directions for Med-Peds reporting • Will be straightforward with the same forms • ACGME sent me a copy of the reporting form when I called them and asked them to; it is literally just the 2 sets put together into 1 form

  39. What about similar milestones? • Some of this is subject to opinion, but about 8-10 are largely the same • What do I mean? • This is split into PC, MK and SBP as one slide and PBLI, Prof, and ICS as second slide • ALERT: COMPLICATED VISUALS!!!

  40. PC, MK, SBP

  41. Bottom line • 22 IM milestones • 21 Peds milestones • 43 total • Of these, 9 have significant overlap, so there are actually 43 minus 9 or 34 SEPARATE milestones to report

  42. Discussion questions from the survey • What about overlapping milestones? • Since the goal is to establish whether residents are progressing and competent, is there a reason not to use the benefit of evaluations from both sides together?

More Related