Using Resident Perceptions to Improve Educational Quality and Accountability

1. Using Resident Perceptions to Improve Educational Quality and Accountability Conference Session:01 Alice Edler, MD, MPH, MA (Educ) – Faculty Fellow Nancy Piro, PhD, Program Manager/Education Specialist Ann M. Dohn, MA – Director, GME Designated Institutional Official (DIO) & Bardia Behravesh, EdD, Program Manager/Education Specialist Department of Graduate Medical Education Stanford University Medical Center

2. Session Objectives • At the end of this session, you will be able to… • Understand how we tap into trainees’ current perception of the quality of GME at Stanford • Understand resident perceptions of factors which influence the quality of their postgraduate medical education • Operationalize the use of survey data for program quality improvement • Be introduced to the use of factor analysis to evaluate survey data

3. Annual GME HouseStaff Survey“Basics” • Annual survey to all HouseStaff • Designed to capture HouseStaff perceptions • Questions are primarily linked to Common Program Requirements • Goal: Early warning and continual program improvement efforts

4. Setting the Stage • 82 Programs (and 82 Program Directors) • 1100 Trainees • 5 Major Affiliate Training Sites

5. Background on our GME HouseStaff Survey • Do we really want to know what they think of us? • Why do we survey? We already have the ACGME survey and annual program evaluations. • What kind of questions do we ask? • What do we do with the responses?

6. Background on our GME HouseStaff Survey • Do we really want to know what they think of us? YES!!!

7. Background on our GME HouseStaff Survey • Why do we survey? • Want residents to be able to speak freely and confidentially • Want to know what is going on across multiple geographic locations and diverse programs • Surveys- instant and easy access (more efficient than Town Halls) • Conserves Residents’ and GME Staff time • Tap into perceptions not queried in program evaluations or the ACGME survey

8. Background on our GME HouseStaff Survey • What kind of questions do we ask? • Information about the Residents • Overall Experience • Program Experience • Program Faculty • Evaluation and Feedback • Training Environment • Personal Wellness • Quality Improvement • Communication and Patient Perceptions

9. Background on our GME HouseStaff Survey • How many questions? • Quantitative and Open Ended / Comments • How do we scale responses? • How do we maintain confidentiality? • How do we encourage residents to respond?

11. Background on our GME HouseStaff Survey • What do we do with the responses? • Explore, analyze, and compare the data (trend by year, program vs institution). • Prepare summary reports for each program • Upload for programs to use the aggregated data for their Annual Program Reviews (APRs)

12. Sample Summary Report

13. Background on our GME HouseStaff Survey • What do we do with the responses? • Quality Improvement • Organizational and Programmatic Change • Data for “Report Cards” • Early Warning System • Chart Progress

14. Lessons Learned • “Charming” • Non-digital HVAC units (cold residents) • 5 Year Accreditation Cycle vs Extremely Unhappy Residents

15. Data Mining • Exploring the Data for New Ideas… • Can residents tell us what elements of education are important to them?

16. Education Quality Educational Leadership Teaching Curriculum

17. Theories of Educational Quality • Skeff • Harris • Cooke, Irby • O’Sullivan, Edler • ACGME • ? The Post Graduate Learner ?

18. Why is the Adult Learner Important? • All learners are important • They are “the starting point, the center and the end” of educational activities (Dewey, 1902) • Adults have a lifetime of past educational experiences • Adults are active participants in their own teaching and learning • Adults are intrinsically motivated

19. Purpose of Our Investigation • To identify elements of the educational milieu that adult learners, in this case, post graduate medical trainees, believe to contribute significantly to the quality of their educational experiences.

20. Overall Goals • To take these perceptions into account to help increase ownership of learning, promote stronger faculty/trainee relationships, and guide educational improvement.

21. Methods • Survey model, with questions from: • Literature reviews • Institutionally relevant themes and accreditation topics • Residents not fellows • 18 Core Residencies • Anonymous • Factor Analysis with Principal Component Analysis

22. Table 1 Core Programs Studied • Anesthesia • Dermatology • Emergency Medicine • Internal Medicine • Neurosurgery • Obstetrics and Gynecology • Ophthalmology • Orthopedic Surgery • Otolaryngology • Pathology • Pediatrics • Physical Medicine & Rehab • Plastic Surgery • Psychiatry • Radiation Oncology • Radiology • Surgery (General) • Urology

23. 32 educationally oriented questions were analyzed using : • Bartlett's Test for adequacy of partial correlations • Determination of Eigen values • All items with Eigen values >0.5 were entered into the factor analysis and then rotated via a Varimax rotation.

24. Response Rates and Demographic Data Results • 88.1% Response Rate ( n = 404) • M:FRatio = 46% : 44% • Missing data 13% • PGY • 1 17% • 2 19% • 3 17% • 4 19% • 5 13% • 6 6%

25. Statistical Results • Significant Bartlett's Test • 15 items with Eigen values >0.5 • All factors positively correlated • 3 principal components identified on Spree plot with 57% of the variance explained • α = 0.889

26. Results • The spree plot suggested the presence of three principal factors. • Fifty-seven percent of the variance was explained by the three factors in this model.

27. Spree Plot

28. Results of the Factor Analysis • Three factors emerged from the initial principal component analysis. • Extraction was completed at ten iterations. • The resultant factor loadings suggested factor one describing curriculum quality, factor two describing teaching quality and factor three describing quality in educational leadership.

30. Component 1

31. Component 2

32. Component 3

33. Summary of Findings • Postgraduate medical trainees can identify 3 key areas of educational quality. • These can be understood as the constructs of curriculum, teaching, and educational leadership. • Our construct findings were validated with answers to questions on the ACGME survey.

34. Curriculum Construct • Our findings validated the students’ understanding of curriculum • One facet among the highest rated was medical ethics instruction • We had just that year added a mandatory on line module for medical ethics education to overcome some departmental insufficiencies in this area. • The lowest rated facet was Systems-Based Practice (SBP) • This was also a low ranking result in the ACGME survey and our own program directors’ needs assessment

35. So What? • It is clear that residents can identify instructional elements and rate their importance and inclusion in their curriculum. • We have also demonstrated that learners who are reared in a competency based era, regard these competencies as critical to their curriculum.

36. Teaching Construct • Our residents identified 3 elements, teaching time both clinically and didactically and, instructor enthusiasm in teaching . • We believe this demonstrated that the teacher and the learner are collaborators in the educational event.

37. Teaching Construct • Irby, in his recommendations for medical teaching in the post Flexner era, suggested we need to “promote relationships with faculty who simultaneously support learners…to create collaborative learning environments” * * Irby, DM, Cooke M, Obrien B, Calls for Reform of Medical Education by the Carnegie Foundation for the Advancement of Teaching: 1910-2010

38. How does this compare with the literature? • Qualitative studies by educational researchers list rapport and communication as the key to excellent teaching. Our findings include the presence of the teacher at formal didactic and conference discussions as a portion of this key element. • Perhaps these formal presentations provide the learner, especially the novice learner, with time to digest and reflect on the materials better than “on the fly” teaching during rounds.

39. Educational Leadership Construct Emphasized Joint decision making in education Respectful atmosphere

40. How does this compare with the literature? • Our findings, like that of Irby et al. suggested that the apprenticeship model of medical teaching need not be abandoned but improved with curricular, instructional and leadership enhancements. • Quality GME education is a complex multi-dimensional endeavor. • Not only must curriculum and instructional methods be taken into account, but also the guidance and quality of leadership. • GME adds an additional layer of complexity.

41. In the professions teacher and learners must come together over the three elements: curriculum, instruction and leadership, but an added unique fourth consideration is omnipresent, the patient.

42. Overall Conclusions • Survey data can be extremely valuable if you are willing to share the data with your stakeholders – in a user friendly way • Be prepared for some surprises • Be prepared to act on your findings – develop action plans • Don’t be afraid to think outside the box

43. Specific Conclusions • Quality education is complex and multidimensional. • Curriculum and instruction are only 2 key elements, the third is educational leadership. • Our findings have corroborated with findings in the general educational literature. • Professional education adds to the complexity by a unique, omnipresent 4th layer: The physician-patient relation.