ASHLEY K. FERNANDES, MD, PHD WRIGHT STATE UNIVERSITY BOONSHOFT SCHOOL OF MEDICINE MARCH 2009

1. A Method of Measuring Outcomes in an Ethics Curriculum A Project for the Harvard Macy Institute's Program for Educators in the Health Professions ASHLEY K. FERNANDES, MD, PHD WRIGHT STATE UNIVERSITY BOONSHOFT SCHOOL OF MEDICINE MARCH 2009

2. The Harvard-Macy Program • 55 Scholars from 16 countries • combines “five major themes: Learning and teaching, curriculum, evaluation, leadership, and information technology.” • Provides knowledge base and skills to enhance expertise in designing and conducting an educational research project • Taking a “leadership role in the educational activities at their home institutions.” • 2 weeks in-residence at Harvard MS in January; 1 week in-residence at Harvard School of Education in May

3. My Project: Purpose • To create an online evaluative tool to test and measure the acquisition and retention of “core concepts” in medical ethics. • To create a series of web-based medical ethics modules which serve three purposes: • (1) to determine whether the core concepts of a preclinical medical ethics course are learned and retained (measurement of outcome) • (2) to emphasize the important content of the course through taking the test itself, as well as through the reintroduction of the topics over time (reinforcement) • (3) to encourage self-directed learning in the medical humanities (expansion of learning).

4. Background • Medical schools in the United States universally accept the idea that a bioethics or medical humanities curriculum is an essential component of medical education, particularly in the pre-clinical years. • Few studies which actually measure outcomes (e.g., changes in behaviors or attitudes, or cognitive retention of core concepts) have demonstrated this assertion in the literature.

5. What are “core concepts?” • “Core concepts” are the “nuts and bolts” of a basic medical ethics education—what we would want our graduates to know on graduation from WSU-SOM • Core concepts are NOT: • Behaviors • Attitudes • Specific moral opinions • Identification of “core concepts” through a literature review—what are medical schools teaching in ethics classes today? • 20-30 “core concepts” have been identified (e.g., autonomy, privacy, palliative care etc.)

6. Core Concepts • Informed Consent • End of Life (EOL): Hospice/ Palliative Care • End of Life (EOL): Euthanasia/PAS • End of Life (EOL): Advance Directives • Confidentiality • Truth-Telling • Medical student ethics (cheating/substance abuse/conscience etc.) • Allocation of scarce medical resources including organ transplantation • Competence/ Incompetence • Access to care • Managed care/Health care economics and delivery • Financial incentives/ conflict of interest • Genetic testing/screening • Reproductive technologies/ Stem cell research • Medical errors and malpractice • Abortion • Role of Ethics committee • Professionalism/ History of Medicine/ Philosophy of Medicine • Ethical Theory/Moral Philosophy • Legal cases/legal issues • Global and Public Health • Bioterrorism • Religion and Spirituality • Respect for Autonomy/Limits to Autonomy • Disabilities • Women/ Minority issues • Brain Death/Coma/ PVS • Geriatrics • Pediatrics/ neonatal care • Misc.

7. The Theory Behind the Project Increasing Complexity of Outcomes Data

8. Kolb’s Learning Styles David Kolb, Experiential Learning, 1984

9. Hypothesis • A medical ethics course, administered in the first year of medical school (and reinforced by online ethics modules in years two through four), will improve scores on multiple choice tests designed to measure the acquisition and retention of core concepts in medical ethics.

10. Experimental Design: The Pre-test • From the list of “core concepts” a multiple-choice and short answer examination bank has been developed • Learning Objectives for each topic were generated prior to the Pre-test design • Students have been given a unique identification number (UIN), which they will use as a login to the ethics test website (for all four years), and which will preserve anonymity for future studies (“w” numbers, with list kept separately).

11. Experimental Design: The Pre-test • Pre-test was given before the 1st SEIM session (25 questions) • https://wisdom.wright.edu/login.html • Class average: 69% • Discrimination factors calculated (range 0-0.44) • Question Difficulty: • 20% Mastery • 40% Intermediate (Moderately Difficult) • 40% Challenging

12. Experimental Design: The Post-test • 50 multiple-choice questions: • 25 identical to pre-test • 25 new, testing the same concepts • Taken March 16, 2009 at 8 a.m.– 60 minutes long • Asked not to study for the exam

13. Experimental Design: Plans for Years 2-4 (pending approval from ‘The Man’) • As a partial fulfillment of the requirement to advance, students will complete additional reinforcement readings and take an examination after completing an online ethics module, testing knowledge of the same core concepts. • Questions will become increasingly complex and decision-oriented • Will work with IT on initial design (approximately 3 full-time work-weeks) • http://oce.sph.unc.edu/phethics/

14. Barriers To Implementation • Approval by FCC, SOM • Requirement • Format • Validating the testing instrument • Statistical analysis is pending • Further development of MCQs—in ethics • Online module design • Maintaining privacy

15. Advantages • Continued reinforcement of an institutional priority (ethics and humanism) • Continued refinement of assessment tools • Keeps ethics in the curriculum and avoids “curricular squeeze” • Self-directed learning • Outcomes data • Is the class improving? Are particular students improving? • Course improvement • Are we teaching certain subjects well? Are there curricular gaps?

16. Some Practicalities and Possibilities • IRB Approval, March 2009 • Internal Medical Education Grant submitted, March 2009 • Timeframe: • Modules (years 2-4) designed and completed by the end of academic year 2010 • Pre- and Post Tests validated by 2010 • Initial (publishable) data by end of academic year 2010 • Need to review demographic data • Compare to SEIM faculty? “Control group?”

17. Selected References • Armstrong EG, Parsa-Parsi R. How can physicians’ learning styles drive educational planning. Academic Medicine 2005; 80(7): 680-684. • Bloom, B, et al. Taxonomy of educational objectives: the classification of educational goals. New York: McKay, 1956. • Campbell A, Chin J, Voo T, How do we know that ethics education produces ethical doctors? Medical Teacher 2007; 29: 431-436. • Collins, J, Writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radio Graphics 2006; 26(2): 543-551. • Eckles R, Meslin E, Gaffney M, Helft P, Medical ethics education: where are we? Where should we be going? A review. Academic Medicine 2005; 80 (12): 1143-1152. • Fox E, Arnold R, Brody B, Medical ethics education: past, present, and future. Academic Medicine 1995; 70(9): 761-769. • Kolb DA, Experiential Learning. New Jersey: Prentice Hall, 1984. • Lehman L, Kasoff W, Koch P, Federman D, A survey of medical ethics education at US and Canadian medical schools. Academic Medicine 2004; 79(7): 682-689. • Smith S, Fryer-Edwards K, Diekema D, Braddock C. Finding effective strategies for teaching ethics: a comparison of two trial interventions. Academic Medicine 204; 79(3): 265-271.