Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior

1. Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement Program (PEP) Pine Grove Behavioral Health and Addictive Services phemphill@forrestgeneral.com

2. FSPHP Presenter Disclosure Slide “Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior” Philip Hemphill, PhD is a paid consultant of Pine Grove Behavioral Health and Addiction Services FSPHP Annual Conference and Meeting Fort Worth, Texas April 23-26, 2012

3. Why use a MSF/360? • Facilitate insight of strengths and weaknesses for an individual. • Enhance organizational cultural change. • Provide summative assessment of performance. • Evaluate potential. • Improve team effectiveness. • Identify training & coaching needs. • Make organizational values explicit. • Measure progress after training & coaching.

4. MSF/360 Essentials • Proven validity & reliability. • Useful in assessing “humanistic” skills. • Weak or no associations between self-rated assessment and external assessment. “…methods such as multisource feedback (360) evaluations may be a necessary next step, particularly when interpersonal, communication skills, or professionalism needs to be evaluated (p. 1101).” Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observed Measures of competence.JAMA, 296 (9), 1094-1102.

5. Key Players in MSF/360 Raters Boss Target Individual Facilitator Vendor

6. MSF/360 Internal Process Feedback Is it consistent with self-perception? Emotional Response Reflection and internal processing of emotion and content. Decision to accept and act, or not Action for learning, change Source: Sargeant et al. (2009) Advances in Health Sc Ed.

7. Relevant Literature Review of Multi-Source Feedback MeasuresThe Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®) • In the late 1950s William Schutz, PhD. developed the FIRO-B theory to aid in the understanding and predicting of how high-performance military teams would work together. • In developing the FIRO-B theory, Schutz began with the premise that "people need people." He used the term interpersonal to indicate any interaction, real or imagined, occurring between people. He used the term need to describe a psychological condition that, if not satisfied, leads to a state of discomfort or anxiety. • He posited that interpersonal needs could be grouped into three categories: Inclusion, Control, and Affection. The FIRO-B model describes the interaction of these three categories of interpersonal need along two dimensions: expressed and wanted. • Awareness Scales and were designed primarily to help individuals with their self-awareness and of understanding their relation to other people (Schutz, 1978).

8. Relevant Literature Review of Multi-Source Feedback Measures Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London,1998) to act upon. Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998). Summative assessment of performance (Newble et al., 1999). Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al.,1997; Facteau et al., 1998). Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998). Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).

9. Development of Disruptive Behavior Instrument • 2004 – Developed list of disruptive behaviors and boundary violation behaviors. Approximately 70 items… • 5 people monitored over 12 months – Instrument too long; complaints from participants; descriptive data…. • 2005- Cut instrument down to 50 items(1 Fctr) • 2008- Cut down to current version • Been using current version for past 4 years…

10. Disruptive behavior is any inappropriate behavior, confrontation, or conflict, ranging from verbal abuse to physical or sexual harassment. Disruptive behavior causes strong psychological and emotional feelings, which can adversely affect patient care. Definition of Disruptive Behavior Rosenstein A, O’Daniel M. (2008). Managing disruptive physician behavior: Impact on staff relationships. Neurology, 70, 1564-1570.

11. Avoids taking responsibility for actions Inappropriately blames others Places other interests above the safety and welfare of community members Suspects, without sufficient basis, that others are exploiting, harming, deceiving, or out to “get” him Seems to have a preoccupation with unjustified doubts about the loyalty or trustworthiness of others Reads hidden, demeaning, or threatening meanings into benign remarks or events Perceives attacks on his character/reputation not apparent to others and is quick to react angrily/counterattack Shows rigidity and stubbornness Displays an unpredictable mood Responds to constructive criticism in an immature/irresponsible fashion Takes too long to requests from others Arrives late Fails to complete paperwork in a timely manner Responds to requests in an avoidant, unreliable, and/or uncooperative manner Displays a sense of entitlement Shows perfectionism that interferes with task completion Behaves in a sexually inappropriate manner (verbally or physically) Lacks empathy Intimidates others or attempts to intimidate others Exhibits physically aggressive or assaultive behaviors Exhibits arrogant behaviors and/or has an arrogant attitude Demonstrates an inflated sense of self-importance Takes advantage of others in order to get what he wants Is reluctant to delegate tasks or to work with others unless they submit to exactly his way of doing things Expresses anger/hostility inappropriately Exhibits passive-aggressive behavior Uses abusive, belittling, condescending, demeaning, and/or threatening language Disruptive Behavior Items

12. CLIENT SIGNS/ UPDATES MONITORING CONTRACT WITH PHP/LAP AND/OR Client signs release allowing for electronic correspondence, and client given monitoring instructions and materials with due date. Client identifies MC, gives MC instructions and rating materials, and with MC identifies approx. 15 raters.. Client completes treatment Client participates in interventions while continuing rating process Rating process is repeated at predetermined intervals until four uneventful reports Ratee email addresses MONITORING RATING FLOWCHART Client attends treatment and rating process is temporarily suspended PHP/LAP recommends interventions No significant impairment is present Collection of surveys online. PHP/LAP recommends intensive/ residential treatment Significant impairment is present and PHP/LAP intervenes Client, MC, and other relevant stakeholders review the feedback report Report produced and emailed to persons indicated on release.

13. Online Survey 33 questions • 28 Likert-type questions regarding ratee’s behavior • 1 question concerning how many hours rater works with ratee • 1 question about the rater’s title • 3 open-ended questions that solicit comments

16. Appropriate Documentation Arrogance Citizenship Compliance Empathy Exploitation of Others Expression of Anger Intimidation Job Attitudes Multicultural Sensitivity Openness to Others Perfectionism Respect for Others Response to Criticism Responsibility for Actions Rigidity Satisfaction with Ratee Sexual Appropriateness Stability of Mood Stress Management Supervision Suspiciousness Tardiness Team Impact Threatening Behavior Verbal Aggression Working with Others Survey Dimensions

17. Disruptive Behavior Instrument • Descriptive Characteristics of Sample: • N = 22 ratees (498 surveys) • Average Age = 38 years • Male = 19 ; Female = 3 • Survey Breakdown by rater: • Self-Report = 53 • Staff = 328 • Supervisor = 37 • Peer = 79 • Number of Times each patient was assessed: • Range – 1-8 times

18. Disruptive Behavior Instrument • Originally hypothesized to be based • on 5 factors • Sexually inappropriate behavior • Sexual Inappropriateness • Sexual/Inappropriate Remarks • Aggression • Aggression: Non-verbal • Aggression: Physical • Aggression: Verbal • Aggression: Written • Self Regulation • Mood Instability • Poor Stress Management • Interpersonal Difficulties • Intimidation • Lack of Empathy • Lack of Multicultural Sensitivity • Lack of Openness to Others • Problematic Perfectionism • Response to Criticism • Rigidity • Suspiciousness • Disruptive Behavior • Arrogance • Blaming Others • Disobeying Rules • Disrespect for Staff/Colleagues • Ease of Working with Ratee • Employee Turnover Intentions • Exploitation of Others • Poor Citizenship • Tardiness

19. Disruptive Behavior Instrument Factor 1 Factor 2 Factor 3 • Results of hypothesized 5-factor loadings; 59% cumulative total variance explained Factor 4 Factor 5

20. Raw Data Eigenvalues, Mean & Percentile Random Data Eigenvalues Ncases 498; Nvars 27; ; Ndatsets 1000; Percent 95 Root Raw Data Means Percentile 1.000000 10.462501 1.456515 1.528056 2.000000 1.836325 1.386788 1.437043 3.000000 1.408055 1.335702 1.380089 4.000000 1.231678 1.291131 1.331221 5.000000 1.178393 1.253103 1.287905 6.000000 .966591 1.216903 1.247532 7.000000 .890236 1.183359 1.212107 8.000000 .805858 1.152611 1.180525 9.000000 .786433 1.121760 1.146968 10.000000 .707731 1.092700 1.117568 11.000000 .671235 1.064358 1.087904 12.000000 .647008 1.037406 1.061681 13.000000 .567647 1.010437 1.033661 14.000000 .518775 .984026 1.007120 15.000000 .514043 .958262 .981678 16.000000 .454498 .932558 .955167 17.000000 .414122 .907291 .929837 18.000000 .395714 .882943 .906811 19.000000 .363423 .857912 .882594 20.000000 .337990 .832012 .855692 21.000000 .312618 .806413 .831047 22.000000 .294869 .780470 .806496 23.000000 .281812 .753717 .780180 24.000000 .268155 .725304 .753368 25.000000 .258453 .695053 .724017 26.000000 .230247 .661538 .694250 27.000000 .195588 .619728 .658277

21. Disruptive Behavior Instrument Factor 1 Factor 2 Factor 3 • Recent analyses reveal that the items are loaded on three distinct factors * Excluded Item 26

22. Disruptive Behavior Instrument

23. Disruptive Behavior Instrument Total Variance Explained

24. Disruptive Behavior Instrument Component Correlation Matrix

25. Disruptive Behavior Instrument Reliability for Three Factors • Poor reliability for items on Factor 3 suggests that a 2 Factor structure may • be a better fit.

26. Disruptive Behavior Instrument2 Factors • Interpersonal Aggression • Aggression: Non-verbal • Aggression: Physical • Aggression: Verbal • Aggression: Written • Arrogance • Blaming Others • Disobeying Rules • Exploitation of Others • Intimidation • Lack of Multicultural Sensitivity • Poor Citizenship • Suspiciousness • Professionalism • Disrespect for Staff/Colleagues • Lack of Empathy • Lack of Openness to Others • Mood Instability • Poor Stress Management • Problematic Perfectionism • Response to Criticism • Rigidity • Tardiness

27. Disruptive Behavior Instrument • Instead of a third factor, items were named “critical items”. • 5 Critical Items: • “I think about quitting or transferring to a different job because of the ratee.” • “Touches or looks at one or more patients/coworkers in an inappropriate/sexually suggestive manner.” • “Makes sexual remarks in the workplace that are inappropriate or tells jokes that makes others uncomfortable.” • “Performs unnecessary exams on female and/or male patients.” • “Demonstrates lax boundaries when prescribing controlled substances, giving out samples, or storing samples of controlled substances.”

28. Disruptive Behavior Instrument • Conclusions/Limitations: • We have 2 main factors and 5 critical items. • Two factors: • Interpersonal Aggression • Professionalism

29. Future Directions: • Measuring and refining Inter-rater Reliability • Measuring and refining Test-Retest Reliability • How is this data utilized? • What are cutoff scores? • Can we track disruptive behavior without treatment? • Can instrument be utilized outside of medicine?

30. We do not deal much in fact when we are contemplating ourselves. Mark Twain