Gargi Roysircar

1. Cultural Diversity Observer Rating Scale: Construct, Convergent, and Discriminant Validity Gargi Roysircar Nicole C. Spanakis, Catherine M. Iacuzzi, Connie L. Bindewald, Susan G. Danielson, Josefina Irigoyen, Michael Brodeur, Mary Quinn, Michiko Ishibashi, Sarah Smith, and Marcelle Abela Antioch New England Graduate School Paper Presented at the 113th Annual Conference of the American Psychological Association Friday, August 19, 2005, 12.00-12.50 p.m., Washington, DC

2. Abstract Twelve trained raters analyzed 480 process notes from 48 doctoral clinical psychology trainees (10 session notes per trainee) who offered one-to-one conversation and mentoring services to culturally diverse individuals in their respective local communities. Raters identified the presence of cultural awareness themes and the use of multicultural interventions in these process notes. Interrater reliability was very high, Cohen’s Kappa .95. The themes and interventions identified by the raters were proposed as the item contents of a measure, the Cultural Diversity Observer Rating Scale (CDORS). Exploratory Factor Analysis using principal components with varimax rotation indicated four factors: Counselor HelpingBehaviors; Counselor Awareness of Client Contexts, Counselor Perceived Barriers and Frustrations, and Diunital Reasoning. The four factors accounted for 37% percent of the variance in the CDORS. Trainees also provided self-report data on the Interpersonal Reactivity Scale, The Hope Scale, Racial Ideology, and the Attribution Questionnaire. Pearson correlations among the CDORS factors and these measures provided evidence of convergent and discriminant validity. Frequency of factors across 10 sessions, as well as in early sessions (1, 2, and 3) versus later sessions (8, 9, and 19) indicated significant differences on two CDORS factors, Counselor Awareness of Client Contexts and Diunital Reasoning. Trainees expressed more Connection thoughts for their clients than Disconnection thoughts across sessions.

3. Description of a Community Activity in Multicultural Training The MulticulturalInteractions Project is an experiential component of a course, called Human Diversity and the Clinical Enterprise, which is taught in a doctoral clinical psychology program in the Northeast by the first author. Trainees are asked to either interview an individual or engage in a service-learning project with an individual who is culturally different from them. This activity is conducted within one’s local community. Trainees’ interviewees are culturally different from them in sociodemographic characteristics such as age, sex, class, culture, race, ethnicity, national origin, sexual identity, religion, and/or ability status. Diversity in multiple areas characterize most of the interviewees. The meetings are a mutual learning opportunity through conversations and friendly support and could have therapeutic effects for the interviewees (called “clients” henceforth). The purpose of these meetings is for trainees to gain awareness of (a) their own assumptions, values, and biases; to be responsive to (b) the worldviews of culturally different individuals; and to develop culturally appropriate intervention skills (Sue, Arredondo, & McDavis, 1993).

4. Components of the Multicultural Interactions Project ●10 meetings (approximately 60 minutes each) in one semester with an individual who is culturally diverse from the trainee ● Weekly process notes submitted to the professor based on these meetings ● Trainee self-reports on measures of personality: Empathy, Hope, Racial Ideology, and Causal Attribution

5. Process Notes ●Trainees are asked to submit weekly process notes to the instructor ●These process notes are meant to be an opportunity for the trainee to examine his or her experience of meeting with a culturally diverse interviewee ●Trainees are asked to focus on process issues (dyadic interactions and trainee intrapersonal reactions) within a meeting session ● Trainees write self-reflections in the process notes. ● For more information on the writing, goals, and outcome of the use of process notes, see: Roysircar, G. (2004). Cultural self-awareness assessment: Practice examples from psychology training. Professional Psychology: Research and Practice, 35(6), 658-666. Roysircar, Gard, Hubbell, & Ortega (2005). Development of counseling trainees’ multicultural awareness through mentoring ESL students. Journal of Multicultural Counseling and Development, 33(1), 17-36

6. Method Sample Forty-eight trainees voluntarily and anonymously contributed the above-mentioned course materials for research. Each trainee contributed 10 process notes for 10 sessions. A total of 480 process notes were analyzed. Forty-eight completed questionnaires were received for each self-report personality measure, Interpersonal Reactivity Index (IRI, Davis, 1980), The State Hope Scale (Snyder, Sympson, Ybasco, Borders, Babyak, & Higgins, 1996), Racial Ideology (adapted from Multidimensional Internal-External Control Scale, Gurin, Gurin, Lao, and Beattie, 1969), and The Attribution Style Questionnaire (ASQ, Peterson, Semmel, von Baeyer, Abramson, Metalsky, & Seligman, 1982). The ASQ has not yet been analyzed.

7. Training of Coders • ● 12 coders held one-hour meetings from January 2004 through July of 2004, from September 2004 through November 2004, and from February 2005 through April 2005 for a total of 50 hours of training. They received training in counselor connection and disconnection themes developed from the qualitative analysis of process notes from 6 counselor trainees in another study (Roysircar et al., 2005). The training involved the 12 raters in understanding and identifying the themes in process notes from trainees utilized for the current study. The meetings also provided an opportunity for peer consultation on the themes and the coding of sample process notes. Some consultations were also held on an individual basis outside of the meetings. • ● See Table 2, Themes for Connection, Disconnection, and Interventions and their Examples. The themes provided the item contents for the proposed measure, Cultural Diversity Observer Rating Scale (CDORS).

8. Rating of Themes ● The coders kept a frequency count of each theme for each process note to determine the presence of the themes and to gain proficiency in recognizing the themes. Coders also tracked client responses, as reported in a process note, which were deemed relevant to a particular theme that was identified in a session process note. ● A rating system was developed to convert frequency counts to a 5-point Likert scale. The scoring reflected the presence or absence of a theme, as well as any response by the client. The scale used was as follows: 1 = theme not present; 2 = one theme with no client response; 3 = the same theme twice without a client response or 1 theme with a client response; 4 = 3 or more times the same theme without a client response or the same theme twice with a client response; and 5 = 3 or more times of the same theme with a client response. ● A separate Likert scale was developed to rate the strength or degree of emphasis placed on interventions that the trainee reported in a process note. The scale developed is as follows: 1 = weak; 2 = mild; 3 = moderate; 4 = strong; 5 = very strong. ● See the Appendix for the CDORS Rating Scale.

9. Interrater Reliability Two raters independently coded each case of 10 process notes. Subsequently, for each case the pair met to discuss their identification of themes and the Likert ratings given to the identified themes. They discussed their respective similar and dissimilar ratings and often through debate reached consensus. The 48 cases were coded by 12 coders, with each coder coding an average of four cases of 40 process notes. The average interrater reliability across the 12 raters (6 pairs) was: Cohen’s Kappa .95.

10. Exploratory Factor Analysis ● An EFA using principal components with a varimax rotation was conducted on 24 items, which were themes generated from trainee process notes for the Cultural Diversity Observer Rating Scale. A criterion of .30 and above item loading was applied as the cut- off for item retention. A four-factor solution emerged according to the eigenvalue greater- than-one rule and the scree plot (36.96% of the total variance accounted for). Inspection of the pattern matrix revealed that all 24 items had the absolute loading of .30 or higher on one, and only one factor (See table 1 for the 24 loaded items). Thus all 24 items were retained. ● The variance contributed by each factor was: 12.84% (Factor 1); 8.43% (Factor 2); 8.03% (Factor 3), and 7.67% (Factor 4). Factor 1 (9 items) was called Counselor Helping Behaviors, which included interventions and counselor communication practices. The second factor (6 items) was called Counselor Awareness of Client Contexts, which included the counselor’s observations of the client’s cultural background, customs, and practices. The third factor was called Counselor’s Perceived Barriers and Frustrations (3 items), which tapped counselor negativity. The fourth factor was called Diunital Reasoning (6 items), which is a process skill in recognizing the reality inherent in two competing, even exclusionary worldviews, that frees the counselor to accept the culturally specific views of the client without having to abandon the counselor’s own worldview. ● See Table 1 for the CDORS item contents and factor loadings

11. Convergent and Discriminant Validity of the CDORS Empathy. Perspective taking was of interest to the study because we wanted to know whether psychology trainees could make empathy-related responses to clients’ references to cultural beliefs and behaviors. Perspective taking is the tendency to spontaneously adopt the viewpoint of another person. In a multifaceted measure of empathy, the Interpersonal Reactivity Index (IRI, Davis, 1980), Perspective Taking is one of four aspects of global empathy. Another subscale of the IRI is Empathic Concern, a dimension of empathy that represents other-oriented feelings of sympathy and concern for the difficulties of others. We expected three factors of the CDORS, Counselor Helping Behaviors, Counselor AwarenessofClient Contexts, and Diunital Reasoning to be correlated strongly with both Perspective Taking and Empathic Concern. Personal Distress, the third subscale of the IRI, was expected to have strong negative correlations with the three previously mentioned CDORS factors because it taps into feelings of personal anxiety and unease in tense interpersonal situations. The fourth IRI subscale, Fantasy, which assesses the ability to empathize with fictitious characters was not of interest for the purpose of the current study. All three IRI subscales (7 items each) had moderate Cronbach’s alphas: .78 (Perspective Taking), .73 (Empathic Concern), and .77 (Personal Distress).

12. Hope. Hope represents expectation for goal attainment. Individuals with high hope approach goals with a positive emotional state and a sense of challenge; these individuals focus on success rather than failure. Low-hope persons experience negative emotional states and a sense of ambivalence; these individuals focus on failure rather than success. The State Hope Scale (12 items) (Snyder, Sympson, Ybasco, Borders, Babyak, & Higgins, 1996) operationally defines hope as goal-directed behavior. It is composed of two factors: agency and pathways. Agency is defined as an individual’s perceived capacity for starting and maintaining the actions necessary to succeed in reaching goals. The pathways factor taps into the individual’s ability to generate routes to those goals. The two subscales were collapsed for the current study and used as a full scale. The Cronbach’s alpha for the full scale was .81. We expected three factors of the CDORS, Counselor Helping Behaviors, Counselor AwarenessofClient Contexts, and Diunital Reasoning to be strongly correlated with The State Hope Scale.

13. Racial Ideology. Race Ideology is a subscale of the Multidimensional Internal-External Control Scale (Multi IE Scale, Gurin, Gurin, Lao, & Beattie, 1969). In recognizing that external, sociopolitical forces operate against individuals who occupy minority status and who face racial, sexual, and economic discrimination, Gurin et al. hypothesized that minority groups may differ in their control beliefs compared to White American men who do not have to contend with such barriers. Race Ideology consists of 13 race-related items. For the purpose of the current study, only 13 external items were used (that is, 13 internal items were excluded) to assess trainees’ beliefs regarding the efficacy of exercising external forces in dealing with racism and discrimination experienced by African Americans (Sodowsky, Kuo-Jackson, Frey, & Corey, 1998). This adapted measure with a Likert scale was called Racial Ideology. Higher scores indicate higher externality. The Cronbach’s alpha for Racial Ideology was .87. We expected the three factors of the CDORS, Counselor Helping Behaviors, Counselor AwarenessofClient Contexts, and Diunital Reasoning to be correlated strongly with Racial Ideology. ●We expected the subscale CounselorPerceived Barriers and Frustrations of the CDORS to have low to nonsignificant correlations with Empathy, Hope, and Racial Ideology, and the three other subscales of the CDORS. ●See Table 3 for the Pearson correlations among all measures and the means and standard deviations of the full sample on these measures.

14. Between Session Analyses of Process Notes ● See Figure 1 for the mean frequency of the CDORS factors in the process notes across 10 sessions. Diunital ReasoningandCounselor Awareness of Client Contextsweremore frequently referred to in the process notesthan Counselor Helping Behaviorsand Counselor Perceived Barriers and Frustrations. ● See Figure 2 for Connection and Disconnection thoughts expressed in process notes across 10 sessions. Trainees made significantly more Connection thoughts than Disconnection thoughts in their process notes across sessions. ● See Table 4 for ANOVAS on frequency of factors between early sessions (Sessions 1, 2, and 3) and later session (Sessions 8, 9, and10). Counselor Awareness of Client Contexts showed significant difference at the .01 level, with trainees being more concerned about knowing the client’s cultural practices in the early sessions than in the later sessions. Diunital Reasoning showed significant difference at the .05 level, with trainees practicing more diunital reasoning in the later sessions than in the earlier sessions. ● See Table 5 for means and standard deviations for frequency of factors in early and later sessions.

15. Future Analyses Another set of process notes contributed voluntarily and anonymously by trainees will be similarly analyzed in order to perform a confirmatory factor analysis on the factor structure derived from the EFA. This will be Study 2. Convergent and discriminant validity will be further investigated in Study 2 by studying the relationships of the CDORS with the Multicultural Counseling Inventory (Sodowsky, Taffe, Gutkin, & Wise, 1994), the Multicultural Social Desirability Scale (Sodowsky et al., 1998), and the Working Alliance (Tracey & Kokotovic, 1989).

16. Conclusion The Cultural Diversity Observer Rating Scale (CDORS) may serve the need expressed in the empirical literature for measures of observed multicultural competencies in counselors and trainees. It could serve as a complement to several self-report measures of counselor multicultural counseling competencies that are currently available. Clinical supervisors could use the CDORS to provide evaluation and supervisory feedback, while counselors could answer the self-report measures for their personal assessment and understanding. Because the CDORS is a short measure (24 items), it serves as a useful outcome measure in studies of counselor effectiveness. The CDORS appears to be psychometrically sound with acceptable reliability and validity. We request supervisors, community clinics, university counseling centers, and instructors of multicultural courses to use the CDORS and test its clinical utility. The first author will be happy to share the CDORS and its codebook upon the completion of the instrument’s development. You may contact the first author at: Gargi Roysircar, Professor of Clinical Psychology, Director, Multicultural Center for Research and Practice, Department of Clinical Psychology, Antioch New England Graduate School, 40 Avon Street, Keene, NH 03431-3552, Ph. (603) 357-3122 ext. 342, FAX (603) 357-1679; g_roysircar-sodowsky@antiochne.edu, http://www.multiculturalcenter.org.

17. Table 1. Exploratory Factor Analysis of the Themes Found in the Process Notes and Cronbach’s Alphas for Factors _____________________________________________________________________________________________ Factor Loading _____________________________________________________________________________________________________________ _____________________________________________________________________________________________________________

18. Note.N = 468 process notes (listwise deletion of missing values). Loadings in bold designate to which factor an item was assigned for subsequent analyses. Int = Intervention; C = Connection; D = Disconnection. Chronbach’s alphas were calculated using ratings across sessions for each factor. Since sessions were situation specific and there was a different case for each session, Cronbach’s alphas may have been lowered with changing data.

19. Table 2. Themes of Connection, Disconnection, Interventions, and Their Examples

21. Examples of Interventions

23. Table 3. Pearson Correlations of the CDOR with Empathy, Hope, and Racial Ideology: Convergent and Discriminant Validity ______________________________________________________________________________________________________________________________ M SD Perspective Empathic Personal Hope Racial Factor 1 Factor 2 Factor 3 Factor 4 Taking Concern Distress Ideology Counselor Awareness Perceived Diunital Helping of Client Barriers, Reasoning Behaviors Contexts Frustrations ___________________________________________________________________________________________________________________ Perspective Taking 24.94 5.09 -- . 97** -.96** .87** .87** .64** .85** .25 .83** Empathic Concern 28.53 3.93 -- -.96** .87** .90** .67** .87** .21 .82** Personal Distress 18.39 4.73 -- -.81** -.87** -.59** -.83** -.20 - .82** Hope 12.31 1.41 -- .84** .71** . 71** .24 .72** Racial Ideology 44.39 10.98 -- .61** .83** .17 .74 Factor 1: 2.34 1.90 -- .46** .09 .36* Counselor Helping Behaviors Factor 2: 7.66 3.89 -- .10 .58** Awareness of Client Contexts Factor 3: 3.84 0.96 -- .39** Perceived Barriers, Frustrations Factor 4: 5.54 2.59 -- Diunital Reasoning ______________________________________________________________________________________________________________________________ Note. * p < .05; ** p < .01 2-Tailed

24. Table 4. ANOVAS on Frequency of Factors Between Early Sessions (Sessions 1, 2, and 3) and Later Sessions (Sessions 8, 9, and 10) Note. *p <.05; **p <.01

25. Table 5. Means and Standard Deviations of Frequency of Factors for Early and Later Sessions Note. Trainees referred less to the cultural practices of clients in later sessions than in the early sessions when they were becoming acquainted with the cultural background of clients (Factor 2; also graphed in Figure 1). There was an increase in trainees’ acceptance of client and counselor competing worldviews in later sessions than in earlier sessions (Factor 4; also graphed in Figure 1).

26. Figure 1. Mean Frequency of Factors in Process Notes Across Sessions FACTORS Diunital Reasoning Counselor Awareness of Client Context Counselor Perceived Barriers and Frustrations Counselor Helping Behavior

27. Figure 2.Connection and Disconnection in Process Notes Across Sessions Connection Disconnection