Ethics Education: Considerations for Development and Evaluation

1. Ethics Education: Considerations for Development and Evaluation Presented at: University of Illinois - Champaign May 2012 Presented by: Dr. Mike Mumford

2. Ethics Education: Needs Assessment • New federal mandates highlight need for effective, validated ethics education programs • Yet, validated instruments or methods for evaluating ethics programs are not applied • Moreover, traditional approaches to ethics lack complexity and empirical evidence

3. Why Evaluate Ethics Education? • Specific federal mandates to do so • Quantify the real value of training program • Ethics education is a loaded gun • It may encourage ethical behavior or it may actually discourage it, and the consequences for either are great • Some (widely-used) programs haven’t worked……

4. Traditional Ethics Programs • Kohlberg-Based Training • Help individuals move from one level of moral development to another • Emphasize others’ rights or philosophical rights of humanity in making ethical decisions • Deemphasize self-interest in ethical dilemmas • CITI Training • Provide instruction on the major ethical guidelines and rules – all declarative knowledge

5. Meta-analyses of Prior Ethics Course Results

7. Ethics Education Evaluation: Federal RCR Mandates • RCR training required in exchange for federal funds • Faculty, graduate students, some undergraduates • No online substitute • Evidence of training effect required • Validated measurement • Failure to comply will have implications for grant and contract awards • Institutions are requiring training for IRB

8. Ethics Education Evaluation: Federal Standards • United States Code and the Code of Federal Regulations requires that all training programs be evaluated • Annual evaluation to demonstrate program effectiveness towards organizational goals • Executive Order 11348: Training for all government sponsored employees • Agency heads must plan, implement, evaluate, and maintain records of training programs • Agency heads must monitor training effectiveness and actively conduct research to improve program effectiveness

9. Ethics Education Evaluation: Quality Control • Systematic evaluation provides insight into programsuccess • What worked and why did it work • Systematic evaluation provides insight into program failure • Is failing a structural problem or a conceptual one • Increases public perception that program has real value • Increases stakeholder “buy-in”

10. Evaluation Methods • Kirkpatrick’s four level model of training evaluation • Trainee reactions: how do trainees like elements of the training • Affect-based; immediate and short-term • Learning: Tests to determine acquired declarative knowledge of training material

11. Evaluation Methods • Kirkpatrick’s four level model of training evaluation • Behavior: Transfer or application of trained knowledge and skills (i.e. trainees’ post-training application of ethics education content) • Results: Impact of training on organizational outcomes (e.g. actual increases or decreases in ethical misconduct, financial gains or losses as a result of training outcomes)

12. Sensemaking Training • Ethics education programs should be grounded in realistic theory of ethical behavior • Traditional models to ethics emphasized: • Moral development • Rational, black-and-white models • Sensemaking training is grounded in EDM

13. Sensemaking Training Overview • Two-day seminar (16 hours of instruction) • 10 modules • 2 out-of-class modules • 8 in-class modules • Key Instructional Objectives • Develop understanding of the ambiguous, complex nature of problems that may be encountered in professional work • Learn strategies that facilitate identification, analysis, and acting within ethical situations

14. Sensemaking Training Content Knowledge of guidelines Awareness of own biases and common errors Model of ethical decision-making (EDM) Situational analysis and interpretation Strategies, or tools, for decision-making Field differences

15. Sensemaking Training Validation • Social & biological sciences graduate student sample • Engineering graduate student and professional sample • Undergraduate REU sample • Pre-post ethical decision-making test • Outcomes examined for pre-post changes • Ethicality of decisions • Strategy use

16. Sensemaking Training Validation * Effects held over 6-month follow-up period

17. EDM Measure Development • SMEs generate field-specific ethical examples • Real-world, ambiguous examples • Equal representation of EDM items across ethical behavior taxonomy • Study Conduct • Data Management • Professional Practices • Business Practices

18. EDM Measure Development • Parallel forms of measure for pre-post format • Test forms counterbalanced • “Pick two” scoring scheme • Pick the “best” two out of possible eight • Code response options for: • Ethicality of Response • Strategies underlying response • Social motives underlying response

19. Example EDM Measure Item(Social Science) Dr. Cedar, a young developmental psychologist, obtained an Early Career Research Grant from the National Institute of Child Health and Human Development to study aggression in elementary school children. Cedar suspects that some children with a certain genetic makeup will be especially susceptible to the effects of television violence. Part of the project requires obtaining a cheek swab for DNA analysis, but interviewing and observing children in the classroom constitutes the major effort. Cedar is collaborating with a well-known senior social psychologist, Dr. Dawson. Given the geographic distance between their universities, the labs communicate primarily via email. • Considering the sensitive nature of the project, Cedar is concerned that parents will be reluctant to allow their children to participate. He is writing the informed consent form and worries that too much detail might discourage participation. How should Cedar handle this issue? Choose two from the following: • Use wording like that of a past study involving similar procedures. • Offer parents the option to contact participants from past data collection efforts to alleviate their concerns. • Contact parents via phone to explain the importance of granting permission for their child’s participation. • Send a follow up letter to non-consenting parents explaining that the risks are minimal. • Mention the minor potential risks in the informed consent along with the benefits of the research. • Describe all possible risks in the informed consent form, no matter how trivial. • Send a handout along with the informed consent form addressing parents’ common questions and concerns. • Since the risks of the study are trivial, deemphasize them in the informed consent form.

20. EDM Measure Validation • Divergent validity evidence: • Narcissism & cynicism (r = -.01 to -.29*) • Exposure to unethical events (r= .02 to -.51*) • Substantive validity evidence: • Meta-cognitive reasoning strategies (r = .12 to .52) • Anti-social behaviors (r = -.07 to -.52)

21. EDM Measure Validation • Criterion-related validity evidence: • Severity of violation (β= -.02 to .35*) • Perceived frequency of violation (β = -. 04 to .04) • Perceived importance of punishment (β = -.01 to .30*) • Severity of punishment recommended (β = -.01 to .26*)

22. EDM Process Strategy Studies • Purpose • Provide evidence for the validity of key EDM processes • Identify strategies underlying those processes • Identify interventions to promote strategy application • 5 key process strategies • Forecasting • Emotion regulation • Framing • Self-reflection • Ethical sensemaking

23. Perceived Requirements for Attaining Goals Perceived Causes of the Situation Professional and Personal Goals RCR Principles & Guidelines Prior Professional Experience Prior Personal Experience Framing Emotions Forecasting Self-reflection Sensemaking Decision EDM Process Strategy Model(Sensemaking)

24. EDM Process Strategy Studies • Sample protocol (Self-Reflection) • Recall past or predict future experiences that are either positive or negative • Consider outcomes or the process of making ethical decision • Complete EDM measure • Sample protocol (emotion regulation) • Train on the use of ER strategies • Induce anger via feedback (anger vs. fear) • Complete EDM measure

25. EDM Process Strategy Studies • Key Findings: Best Strategies • Forecasting • Analysis of critical causes improved forecasting & EDM • Forecasting critical consequences improved EDM • Time pressure didn’t matter • Self Reflection • Reflection on positive events and process increased EDM • Using reflections to predict future ethical events improved EDM • Emotion Regulation • Anger inhibited EDM, moderate fear facilitated EDM • Reappraisal buffered anger – was an effective technique

26. Case-Based Learning Studies • Purpose: • Cases are frequently used in ethics education, yet little is known about development of cases • Little is known about methods for case delivery • Three rounds of studies: • Case content - finished • Process methods (strategies for delivering cases) – in progress • Content & process interactions - 2013

27. Case-Based Learning Studies • Case Content: • Character affect & power dynamics • Codes of conduct & forecasting • Critical causes & outcomes • Social context & goals • Process methods: • Elaboration & self-generation of cases • Sequential case presentation vs. case comparison • Alternative scenarios & outcome evaluation • Incremental case building & forecasting • Note-taking & review

28. Case-Based Learning Studies - Findings • Content Studies: • Character emotions enrich cases  increase in knowledge acquisition & transfer of ethical principles • Highlight codes of conduct  “ “ • Include causal information and potential outcomes • Process Studies: • Ask individuals to elaborate on cases • Have individuals compare multiple cases • Providing alternative case endings confuses readers

29. Alternative Methods of Evaluation • Knowledge measures • Declarative Ethical Knowledge Measure (DuBois and Colleagues (2012) • Knowledge of codes of conduct and regulations • Average Cronbach’s alpha = .71 • Correlates highly with EDM measure (r = .40, p < .01)

30. Alternative Methods of Evaluation • Trainee reactions • Trainees in sensemaking training have reported high levels of satisfaction with training activities and content (M = 5 - 6 on a 7-point scale) • Change in ethical conduct • Distal evaluation • Track ethical violations over time

31. A Cautionary Ending • It is dangerous not to evaluate ethics training programs • We expect that ethics programs only help, but in reality they can do more harm than good • Success or failure (and the consequences) grounded in: • Effective models for understanding ethical behavior • Multi-faceted, comprehensive program evaluation