New Directions in Assessment

1. New Directions in Assessment Stuart Goldman MD Harvard Macy 2009

2. Goals • To reconsider current paradigms of assessment their strengths and limitations • To consider the possibilities of complementary models • To review one such model- the Educational Kanban or EK

3. Assumptions and Values I • What are the basic assumptions, premises and values that we have as clinical educators about assessment? • How do these successfully inform and guide what we do? • What has worked for you as an educator providing assessment?

4. Assumptions and Values II • Place yourself in the position of either a medical student or house officer • Form small groups of 3-4 • What are the assumptions and values we have from that perspective? • Please generate a list and consider how your “assessment” experiences have worked or failed and how “assessment” might be improved?

5. Case Discussion –Sonja I • Sonja is PGY-IV at the end of her first year of training in a child psychiatry program. She graduated from a general psychiatry program with solid letters of recommendation. She is sitting down with you, as her training director, after the March department wide resident review. She has finished 2 of 3 of her principle first year rotations. At the meeting you were dismayed to hear that both of her two primary attendings felt her work was unsatisfactory.

6. Case Discussion –Sonja II • More careful review, at the meeting, revealed the Sonja was professional and collaborative in her daily presentation and well liked by her patients, but, despite “best efforts”, her grasp of the “big picture” of patients was markedly lacking. She was said to have a limited fund of knowledge, trouble with interviewing, diagnosing and setting up a treatment plan, and was often over-identified with her patients. Both attendings had the same set of concerns and noted no progress. Both claim to have reviewed this with Sonja.

7. Case Discussion –Sonja III • At your meeting, you begin with asking Sonja “How does she think things are going?” She responds that she really thinks things are going pretty well, in fact “great.” She is happy in the program and feels the work was initially challenging but is now progressing nicely. At first gently, then more firmly, you raise the concerns of her attendings. She claims to have never heard this before (neither rotation form was co-signed by her). You tactfully press this issue and she says that “she respectfully disagrees” and that your concerns are “unfounded and just wrong.” Now what, you wonder?

8. Some Basic Problems in “traditional/typical” assessment I • Balancing promoting learning and growth with the need to demonstrate competence • Fences verses ladders • Tension between cost ($,time) and effectiveness • Failure to integrate adult learning theory • Adult learners need to be actively involved in the learning process from initial self-appraisal to setting goals, developing mastery and to participation in the evaluative process all set in context. Give adults control, do not infantilize them

9. Some Basic Problems in “traditional/typical” assessment II • Internal verses external motivation • Lack of real time feedback • While clinical education is often 1 to 1, the specific learner in the specific rotation is rarely considered • Little continuity from one rotation to the next • Problem of educational “Ground Hog Day”

10. Approaches to Quality Control and Promoting Production (Learning) • Top Down – General Motors • Production goals, material, manufacturing, specific jobs all designed (with best intent) from the top down- SUVs made sense until…… • Material, manufacturing and market conditions are not rapidly fed back into goals or the production process • Most traditional educational models and assessments are typically structured this way and this has real value, but can there be more/different approaches ?

11. Bottom Up - Toyota • Assess the market- what are the market “pulls ?” Prius • Production goals, materials, manufacturing and jobs all centered on real time specific experience: evaluated and adjusted to insure production success. • The product needed is the product made • Goal is to continuously improve and succeed • Shift from why aren’t you doing it right to how can we help you get it right?

12. Bottom Up II • Rapid adjustment through propagation of need - the Kanban (card)- concrete signaling device that “pulls” supplies • The Kanban’s focus is process-oriented promotion of the worker and systems’ efficiency and quality • Conservation of capital by having the appropriate amount of materials and support needed to accomplish the task at hand in real time. • Worker’s conservation of labor by doing the specifically needed work. • Can this be integrated into educational assessment?

13. Educational Kanban I • A “pull” approach to assessment in clinical settings grounded in both adult learning and effective manufacturing • Computer based document –“owned” by trainee and not in their “permanent record” • Emphasis on tracking continuous improvement that is owned by and travels with the trainee addressing educational process and systems problems

14. Educational Kanban II • Begins with cross links/activation of prior experiences through a meeting between attending and student- Process is: • Student reviews EK prior to meeting • Review jointly the EK to date— • Attending provides rotation and program specific goals that are C/W program and ACGME • Blended goals for this specific student on this specific rotation, based upon their specific experience/progress and need • All meetings are student led.

15. Educational Kanban III • Takes advantage of 1 to 1 clinical teaching environment-developing specific plan • Meet regularly to monitor progress • Readjust as needed reflecting real time change • End with review, update Kanban • Carry the EK as they move on to next rotation to pull the appropriate educational efforts on the next rotation. • Complements the still needed other assessments

16. EK Pushback • Takes more time- both in the initial outlay and real time feedback • But ultimately teaching what is specifically needed will conserve resources for both educators and learners • Shifts from fences to ladders • Values trainees as adult learners • Real time correction • Elimination of “educational ground hog day” • Readily dovetails with program, rotation, AAMC/ACGME objectives, shifts ownership, decreases infantilization

18. Resident A is a generally good first year child resident who comes from a well-regarded adult program. On her first rotation (Inpatient) she was noted to be hard working and conscientious. She related very well to patients, but had some problems with parents. In meeting with her inpatient supervisor, she realized that, parents who just didn’t “get it” pushed “her buttons”. Her knowledge of development and interviewing was seen as lagging somewhat behind her peers. She was off to a good start with case formulation, but she attributed too much of the problem to parental deficits. This might be the meeting with her CL supervisor as she moves from her first to second rotation.

19. 1. Patient Care • Establish/sustain therapeutic alliance­­­­­­­­­­­­­ __Ability to engage pt/ family and to collaborate over time, working towards the agreed upon goal. Must understand broad alliance concept and operationalization with pt/family • Clinical interviewing including; History, MSE.Developmentally informed interviews, eliciting all needed data, and organizing it appropriately • Request/assimilate ancillary data sources to support care planContacts schools, professionals, understands psychological testing • Multidimensional diagnosis_ Understands DSM multi-axial DX, Can frame clinical profile from; biological, developmental, familial, social perspectives

20. Multidimensional Formulation and Treatment PlanUtilizes bio-pscyho- social model to explain (formulate) case and can establish formulation-based treatment plan • Appreciate/apply appropriate treatments; These include crisis management, CBT, Play, Family, Dynamic, Supportive, all as indicated, (will vary from service to service) • Case management skills Manages cases appropriate to context, including collaboration with pt/family, professionals, treatment planning and record keeping. Appreciates good management=good care.

21. Reflection: I Patient Care: I am having some difficulties with families who mistreat their children. I also would like to have a clearer picture of how to interview little ones. I felt that the quick overview of development didn’t give me all the foundation I needed and would like to practice. • Goals: To be more open to the patient/family’s viewpoint, understanding it from their perspective. Work on my interviewing skills, and my CBT skills. Maybe I can do some more developmental reading

22. Comments/Plan: Collaboration with families and professionals is key here on CL. Lets jointly see some cases and then we can discuss what you observe and I have done. We also will focus on the “developmental” aspects of each case and I can give you more reading. CBT informed interventions are common here and you will have the chance to practice. Let’s plan to have you work on learning patient relaxation techniques as a starting point. Finally CL is a great place to observe and be observed and we can work on continuing to build your interviewing skills.

23. Educational Kanban • The EK is one approach to supporting adult learners to own their education, focuses on continuous improvement, minimizes the adversarial elements of assessment, complements/dovetails with other assessments and empowers them to grow. • It will counter the endemic “duck and cower” approach that many students have to assessment embracing the maxim “that great teachers make their students brave *” • Professor Roland “Chris” Christensen