Transformation of a Clinical Assessment System: Involving All Faculty Members as Stakeholders

1. Transformation of a Clinical Assessment System: Involving All Faculty Members as Stakeholders Wilda Guzmán DMD, MPH University of Puerto Rico School of Dental Medicine

2. Objectives • Present an innovative venture in the clinical assessment system of the University of Puerto Rico School of Dental Medicine. • Demonstrate the use of a model in organizing a change process.

3. Rationale of Project • The clinical assessment system of the University of Puerto Rico School of Dental Medicine: • in use for over 15 years • includes daily evaluations based on rubrics, discipline based clinical exams, and numeric procedural requirements (checklist) • recently integrated a computer program (ACO) http://aco.rcm.upr.edu/aco

4. Rationale of Project • Perception: • System not used optimally. • Some assessment methods not competency based. • Faculty resist the use of ACO. • Faculty evaluate the students based on personal clinical experience and preference. • Majority of the evaluations consist of the highest numbers of the scale used.

5. Rationale of Project • Recent developments at national level: • Updated set of outcomes “Competencies for the New General Dentist” (2008) • Revision of accreditation standards • Increased emphasis on cultivating student’s capacity for critical thinking and self-assessment

6. Description of Project • A transformation of the clinical assessment system has been initiated with the hope of aligning strategies with competency based curriculum (accreditation standard). • The transformation process is attempting to involve all the faculty as stakeholders. • The process has been organized according to Kotter’s Eight Steps Change Model.

7. Question • Does involving all the faculty as stakeholders in the transformation of a clinical assessment system result in: • Improved effectiveness • Appropriate and increased use by the faculty • Increased satisfaction of the students?

8. Kotter’s Eight Steps Change Model Leading Change: Why Transformation Efforts Fail – John Kotter http://www.naph.org/NAPH/2008_Fellows/Leading_Change_-_Why_Transformations_Efforts_Fail_-_John_Kotter.pdf

9. Kotter’s Eight Steps Change Model • Establishing a Sense of Urgency • Forming a Powerful Coalition • Creating a Vision • Communicating the Vision • Empowering Others to Act on the Vision • Planning for and Creating Short Term Wins • Consolidating Improvements and Producing Still More Change • Institutionalizing New Approaches • Legend: • Completed • In Process • Planned Legend: Completed In Process Planned Legend: Completed In Process Planned Legend: Completed In Process Planned Legend: Completed In Process Planned Legend: Completed In Process Planned

10. Establishing a Sense of Urgency • American Dental Education Association (ADEA) white papers • Changes in dental education • Faculty meetings • Survey to faculty (needs assessment) • 72% participation • 5 components of present clinical assessment system • Lack of knowledge and confusion among faculty in all areas addressed in survey

11. Baseline data Establishing a Sense of Urgency

12. Establishing a Sense of Urgency • Analysis of present system components • Syllabi of clinical courses • Presentation to Dean, staff and to faculty • Results • Lack of knowledge about the different components of the clinical assessment system • Lack of motivation • Resistance to use of system • Some documents and examinations need improvement to focus on competencies • Need for critical thinking, evidence based dentistry, adequate feedback and multiple assessment methods focused on student learning

13. Forming a Powerful Coalition • UPR ADEA Commission on Change and Innovation (CCI) Liaison: 4 faculty members and 3 administration members • Weekly meetings • Gaining Dean and staff support

14. Creating a Vision • The clinical assessment system of the UPR School of Dental Medicine will be fully understood and thoroughly used by both faculty and students to improve student performance throughout the curriculum. • The system will: • utilize well defined criteria and instruments, • be based on the School’s competencies, • promote critical thinking, self-assessment and self-directed learning, • be a model for use in the pre-clinical dental classes of the undergraduate curriculum, • facilitate faculty calibration.

15. Creating a Vision • Creation of Logo

16. Communicating the Vision • Logo and vision in all communications and presentations • Creation of UPR CCI webpage • http://dental.rcm.upr.edu • http://dental.rcm.upr.edu/adea-cci.html • Role modeling

17. Empowering Others to Act on the Vision • Annual Faculty Workshop 2008 based on transformation • Review of findings • New assessment methods • Rubrics • Transformation of system • Pilot project

18. Context-free Multiple Choice Questions (MCQ) Case-Based Multiple Choice Questions(MCQ) Essay Oral Examinations Research Reports and Presentations Critical Appraisal Tasks Portfolio-driven assessments Requirements Clinical competency examinations Computer-Based Simulations Laboratory Exercises (Practical) Chart-Stimulated Exams Objective Structured Clinical Examination (OSCE) Triple Jump Exercise (TJE; or 3J) Global evaluations Daily evaluations Students’ self-assessment of performance Assessment Methods

19. Empowering Others to Act on the Vision • Developed in workshop: • Ideas for assessing competency development • Rubrics for new methods Faculty Workshop July 2008

20. Empowering Others to Act on the Vision – Pilot Project • Comprehensive Care Clinical Course for Senior Students: • Needed fewer modifications • Faculty coordinating teams more open minded • Dean of Clinic supportive • More control than other courses

21. Empowering Others to Act on the Vision – Pilot Project • Competency based trimestral evaluations to substitute daily evaluations that were discipline based • October, January, April • Approximately 10 faculty members involved in supervision • Round table assessment of each student • Rubric used • Summary evaluation form signed by all faculty members • Coordinator meets afterwards with each student to discuss results and develop plan to address deficiencies

22. Evaluation Form Legend A: Above Expectations M: Meets Expectations B: Below Expectations N: Not Worked

23. Empowering Others to Act on the Vision – Pilot Project • Changes: • ACO involves only Completion of tickets • Emphasis on meaningful daily oral feedback • Barrier of grade eliminated • No numeric procedural requirements, but completion of cases • Overall production considered in grade

24. Empowering Others to Act on the Vision – Pilot Project • Electronic Portfolio • Personal reflection based on competency development • Special report with treatment decisions based on evidence • 3 cases from diagnosis to completion • Self assessment of cases based on standards of care document • Oral presentation

25. Empowering Others to Act on the Vision – Pilot Project • Competency tests • Complete reorganization • Preparation of rubrics • Improvement of methodology • Incorporation of new assessment methods • Based on competencies

26. Empowering Others to Act on the Vision • Pilot Project presented to faculty and students • Selection of group of faculty as leaders for pilot project • Elimination of obstacles • Changes in computer program • “No no” faculty are minority • Better environment for feedback • Simplification of forms • Responsibility of assessment is collective • Training to prepare the faculty and students

27. Planning for and Creating Short Term Wins • Pilot Project begun • Trimestral evaluations • Well received by faculty and students • Documents are being prepared for use next year (complete implementation) • Model is being considered for 3rd year students • Communication of wins to faculty and students as they occur • Reward involved faculty • Recognition

28. Consolidating Improvements and Producing Still More ChangeExtension to other courses Institutionalizing New Approaches Training of new faculty

29. Findings to Date • Kotter’s model is very useful for organizing the process. • Gaining support from the administration and faculty is essential. • CCI Liaison has been effective in producing discussions among faculty about the needs for change and new assessment methods. • Continuous training and faculty workshops are essential in dental education.

30. References • Albino, J., et al. Assessing dental student’scompetence: bestpracticerecommendations in the performance assessmentliterature and investigation of currentpractices in predoctoral dental education. J DentEduc 2008; 72: 1405-1435. • Guskin, A., Marcy, M. Dealingwiththefuturenow. Change 2003; July/August: 10-21. • Haden, ADEA CCI. The dental education environment. J Dent Educ 2006; 70: 1265-70. • Kotter, J. Leading change: why transformation efforts fail. HBR 1995; March-April: 59-67. • Kotter, J., Rathgeber, H. Our Iceberg is Melting. St.Martin’s Press 2006. First Edition. • Pyle, ADEA CCI. The case for change in dental education. J Dent Educ 2006; 70: 921-4.

31. Questions? wguzman@rcm.upr.edu

32. Thank You!